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FOREWORD 



Rapidly accelerating technological advances in all phases 
of our lives, but especially in the workplace, have created an 
increasing need for vocational education students to understand 
fundamental principles about the work they may pursue. The 
development of curriculum materials for teachers and students is 
an essential step in the process of assuring that students acquire 
an understanding of technological principles. 

The Center for Occupational Research (CORD), Waco, Texas, 
and the Agency for Instructional Technology (AIT), Bloomington, 
Indiana, have been working with a consortium of State agencies for 
vocational education to develop curriculum materials dealing with 
the principles of technology as they apply to the mechanical, 
fluidal, thermal, and electrical energy systems of modern equip- 
ment. CORD and AIT in conjunction with the participating States 
designed formative evaluation processes to be used during the 
2-year effort. The National Center for Research in Vocational 
Education became involved with the Principles of Technology effort 
during the last part of the first year of implementation of the 
curriculum materials. Given this entry point, the role of the 
National Center has been one of reviewing the already operation- 
aliped formative evaluation design and making suggestions for 
improving the formative evaluation. 

The National Center expresses its appreciation to the CORD 
and AIT staff members who were most cooperative in this effort. 
Evaluation managers at AIT, Bill Johnston and Jim Shea were 
especially helpful. Dr. Tim Wentling, Professor, University of 
Illinois, and Phillip Rollain, Project Coordinator, North Carolina 
Department of Public Instruction, served as panel members to 
review the National Center formative evaluation suggestions. The 
National Center is also appreciative of the reviews provided by 
Dr. John Washburn, Manager, Research and Development, Illinois 
Department of Adult Vocational-Technical Education. 

The National Center is indebted to the staff who worked on 
this task. The study was conducted in the Evaluation and Policy 
Division, Dr. N.L. McCaslin, Associate Director. Dr. Floyd L. 
McKinney, Senior Research Specialist, served as project director, 
and Alan Kohan served as Graduate Research Associate. 

Funding for the study was provided by the Office of Voca- 
tional and Adult Education, U.S. Department of Education. 

Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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EXECUTIVE SUMMARY 



The focus of this task was to improve the formative evalu- 
ation design for the Principles of Technology (PT) curriculum 
materials. The Principles of Technology curriculum materials were 
designed to prepare secondary students for technical careers in 
the complex and rapidly changing technological field. The cur- 
riculum materials were developed by staff of the Center for Occu- 
pational Research and Development and the Agency for Instructional 
Technology working with a consortium of 35 State and Canadian 
vocational education agencies. The curriculum is comprised of 14 
units, with each unit dealing with one principle of technology as 
it applies to the mechanical, fluidal, thermal, and electrical 
energy systems of modern equipment. 

This task was a 12-month effort that began on January 16, 
1985. Thus, National Center staff entered the Principles of 
Technology 2-year formative evaluation effort at approximately the 
midpoint of the PT evaluation time line. The formative evaluation 
design for units 1-7 was developed and implemented prior to the 
entry of the National Center in the formative evaluation effort. 
Given this set of circumstances, National Center staff reviewed 
the formative evaluation activities already being conducted by 
AIT/CORD and made suggestions for improving the evaluation effort. 
Minor changes were suggested in the formative evaluation instru- 
ments used by teachers and students. The major suggestion for 
improving the formative evaluation was the use of case studies. 
The case studies provided the curriculum developers with greater 
depth of information about the enormously complex settr.ng in which 
local schools were attempting to implement PT. 

The following recommendations ari; based on staff involvement 
with the design for the formative evaluation of the Principles of 
Technology curriculum effort, previous evaluation efforts, and 
reviews of relevant literature: 

o The designing of a formative evaluation effort should be 
done concurrently with the development of the curriculum 
and the development of strategies for field testing the 
curriculum. 

o The purpose and nature of formative evaluation should be 
thoroughly explained for all concerned parties so that 
expectations for results do not overburden or undercut the 
evaluation effort. 

o Extensive use of the telephone or other means that permit 
two-way communication should be made by the curriculum 
developer/evaluator and the curriculum user (teachers* 
students). However, this should not be construed as an 
acceptable substitute for on-site, in-depth observations, 
interviews, and document or record reviews. 



ix 



10 



CHAPTER 1 



INTRODUCTION 



This report describes the involvement of the National Center 
for Research in Vocational Education with the Center for Occupa- 
tional Research and Development (CORD), Waco, Texas, and the 
Agency for Instructional Technology (AIT), Bloomington, Indiana, 
concerning the formative evaluation of the Principles of Tech- 
nology curriculum materials. In this chapter a brief overview of 
the Principles of Technology effort is presented so that the 
reader will have a better understanding of the National Center's 
involvement in the formative evaluation. 

The Principles of Technology curriculum is designed to pre- 
pare students for technical careers in the complex and rapidly 
changing technological field. Based on the Unified Technical 
Concepts (UTC) course developed by CORD, the curriculum is com- 
prised of 14 units, each addressing one principle of technology as 
it applies to the mechanical, fluidal, thermal, and electrical 
energy systems of modern equipment. Because the design and opera- 
tion of modern equipment undergo constant change, technicians, if 
they are to adapt to the changing workplace, need to know the 
fundamental principles on which such equipment operates. Then, 
once technicians understand the principles on which their work is 
based, they can apply these principles to new work situations as 
the need develops. 

The development of the Principles of Technology curriculum 
was a cooperative effort comprised of 35 State and provincial 
vocational education agencies in association with AIT and CORD. 
Inherent to the development process for the Principles of Tech- 
nology curriculum was the assumption that this process is based on 
approximate and imperfect knowledge. Therefore, verification of 
the curriculum during its development process was essential to 
ensure that the final product was internally consistent and effec- 
tive. However, internal and external constraints necessitated 
compromises between ideal evaluation procedures on one hand and 
feasibility on the other. The problem was to design a formative 
evaluation for the Principles of Technology Curriculum that could 
be accomplished with a relatively small investment of time; 
provided specific revision feedback about the effectiveness of the 
overall learning system, including the printed materials, lab 
activities, learning kits, displays, and audiovisual materials; 
did not confuse or frustrate trainees during their coursework; and 
provided quick results. 
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Objectives 



The National Center had two task objectives: 

o To design a formative evaluation for the Principles of 
Technology curriculum materials 

o To assess the implementation of the evaluation design 

The target audience for this task was the AIT evaluation 
staff. The intended use of the formative evaluat ; >n suggestions 
generated by the task staff was for improvement of tne formative 
evaluation of the PT curriculum materials. 



Pri nciples of Technology 

History 

The content of the Principles of Technology curriculum was 
based on the UTC instructional model developed about 7 years ago 
by CORD for use at the postsecondary, associate degree level. 
Perhaps the most interesting aspect of the UTC curriculum was the 
manner in which the key elements of technology — elect rical, 
mechanical, fluidal, and thermal principles — were presented- 
Rather than being presented as isolated concepts, the key elements 
of technology in the UTC curriculum were presented together as 
analogous phenomena. Moreover, the UTC curriculum was oriented 
toward preparing students for high-technology occupations bv 
providing a broad-based background in applied physics. Industrial 
applications of the UTC principles were emphasized throughout the 
curriculum. 

The need for a similar system of instruction at the secondary 
level is being acknowledged by state departments of education. 
The increased requirements in math and science for graduation from 
high school have made it more difficult for high school students 
to complete a traditional vocational education program. Further- 
more, the increasing emphasis on technological literacy and a 
somewhat decreasing emphasis on specific job-skill training at the 
secondary level have added legitimate pressures for changing the 
focus of secondary vocational education programs. By April 1983, 
the increased interest in having the UTC curriculum adapted for 
implementation at the secondary level led to the formation of a 
consortium of about 30 States to initiate development of such a 
curriculum. From an economic standpoint, the consortium, in 
association with CORD and AIT, was a feasible strategy for the 
initial development of the Principles of Technology curriculum. 
The prosj ictus for the project was issued in June 1983, and the 
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initial development work began in Novemoer 1983. Currently, the 
consortium membership is comprised of a total of 35 States and 
Canadian provinces and represents an investment of approximately 
$2.5 million for the development of the Principles of Technology 
curriculum. 



Q^TView of Initial Formative Evaluation Process 

In cooperation with CORD, AIT was responsible for w.he forma- 
tive evaluation of the Principles of Technology curriculum. CORD 
was responsible for developing the unit objectives ana printed 
materials. Video programs were developed by AIT. An outside 
review team, comprised of eight members (appendix A), reviewed all 
materials before pilot testing. Consortium agencies also received 
curriculum materials for review prior to the pilot tests* 

An important part of the developmental process was a pilot 
test of each instructional unit in actual classroom settings. The 
primary purposes of the pilot test were to determine how well the 
materials were working and to identify specific problems with the 
materials. Each consortium agency was allotted two pilot-test 
sites. Test sites and teachers, however, were selected by the 
respective States and provinces. AIT had no control over the test 
site selection criteria. 

Pilot-test evaluation materials were comprised of a pretest 
and a posttest, a student attitude questionnaire, and a detailed 
teacher questionnaire (appendix B) . in brief, teachers adminis- 
tered the pretest prior to teaching the unit. While the unit was 
being taught, teachers recorded their reactions to the unit on the 
detailed teacher questionnaire. At the conclusion of the unit, 
teachers administered the posttest and student altitude question- 
naire. The pretest, posttest, student questionnaire, and teacher 
questionnaire were scored and analyzed by AIT staff. 

In May 1984, a 1-week tryout of the pilot-test procedurss was 
conducted a^ a vocational school in Tulsa, Oklah^ua. A prelimi- 
nary version of the materials for the subunit "Force in Mechanical 
Systems" was used in the tryout. One teacher and nine students 
participated in the tryout. Observers were present during the 5 
days of instruction. Overall, AIT staff concluded that the 
Oklahoma tryout was useful in identifying positive indications 
about the curriculum and evaluation procedures as well as in 
identifying some legitimate concerns. 

In June 1984, a Principles of Technology teacher orientation 
meeting was held in Dallas, Texas. Approximately 60 teachers 
participated. The primary purpose of the workshop was to orient 



pilot-test teachers to the Principles of Technology curriculum and 
its instructional plan and to explain formative evaluation data 
collection activities and procedures for the upcoming pilot-test 
year* 

Instructional units 1-7 were pilot-tested from September 1984 
tc June 1985. Of the original 60 pilot-test teachers, about 30 
teachers completed all of the evaluation forms during the first 
pilot-test year. Although there was a variety of reasons why test 
site schools did not fully participate, the overall reason 
copeared to be a problem in obtaining the necessary instructional 
equipment and/or school funds to purchase program start-up 
equij^nent. Nevertheless, approximately 400-600 students 
participated in the first-year pilot test. 

An overview of the project time line for the curriculum 
material production and development is presented in appendix C. 
Instructional units 1-7 were released in their final form during 
the period July-November 1985. The pilot-testing of units 8-14 
began in September 1985, and they were scheduled for final release 
in September 1986. 



CHAPTER 2 



FORMATIVE EVALUATION FOR CURRICULUM DEVELOPMENT 



Two essential activities in considering suggestions for 
improving the formative evaluation of the Principles of Technology 
curriculum materials were the following: (1) identifying relevant 
information needs and (2) reviewing alternative approaches that 
might have desirable features for consideration. The following 
sections describe task efforts for these two activities. 



A Classification of Trleal i nformation 

Sanders and Cunningham (1973) categorized three major sources 
of information as being essential for the formative evaluation of 
educational materials. (See table 1.) Although the categories 
were not mutually exclusive or exhaustive, Sanders and Cunningham 
noted that the important point was not the classification scheme 
in and of itself, but that no source of information should be 
overlooked, given project resources and constraints. 



Selected Alternative Approach es to Formative Evaluation 
According to Dick and Carey (1978), 

formative evaluation is the process used to obtain data 
for instructors to use to increase the efficiency and 
effectiveness of their instructional materials. The 
emphasis in formative evaluation is on the collection of 
data in order to revise the individual materials, to 
make the materials as effective as possible. (p. 159) 

Dick and Carey noted that when the instructional materials are in 
their final form, "other people may collect data to determine 
whether the materials should be used in a particular setting or 
whether they are effective as claimed" (p. 159) . 

Sarapin (1981) noted that the foundation for formative evalu- 
ation for curriculum development and improvement was based upon 
evaluation methodology reported by Scriven (1967), Stake (1967), 
^cufflebeam (1967, 1969), Stufflebeam et al. (1971), and Provus 
(1969). Several scholars (Abedor 1971; Sanders and Cunningham 
1973; Baker and Alkin 1973; Dick and Carey 1978, 1985; and Sarapin 
1981) have conceptu Uized formative evaluation models and pro- 
cedures. From these conceptualizations, various approaches to 
formative evaluation have emerged. Brief descriptions of these 
approaches that were identified as being pertinent to the Princi- 
ples of Technology project are highlighted in the following sub- 
sections. 
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TABLE 1 



A CLASSIFICATION OP INFORMATION NEEDS IN 
FORMATIVE EVALUATION 



I. Internal information 

A. Descriptive information 

1. Physical specification 

2. Rationale, goals, and objectives 

3. Content 

4. Other 

B. Critical appraisal 

1. Author (developer) 

2. Experts (subject matter, media, psychologists, and 

so forth) 
3 Students using the materials 

4. Teachers using the materials 

5. Relevant others 

II. External information 

A. Assessment of the effects of the materials on student 
behavior 

1. Achievement 

2. Attitude 

3. Skill 

4. Interest 

5. Commitment 

6. Other 

B. Assessment of the effects of the materials on 
behavior 

1. Attitude 

2. Interest 

3 . Commitment 

4. Competency 

5. Teaching strategy 

6. Other 
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TABLE 1 — Continued 



C. Assessment of the effects of the materials on the 
behavior of relevant others 

1. Parents 

2 . Administrators 

3. Teachers not using the materials 

4. Students not using the materials 

5. The community 

6. Other 

III. Contextual information 

A. Student characteristics 

B. Teacher characteristics 

C. School characteristics 

D. Community characteristics 

E. Curricular characteristics 

F. Other relevant elements in the learning environment 



SOURCE: Sanders and Cunningham (1973, p. 218). 

Clinical Approac h (One-to-One) 

Lowe r Thuirston, and Brown (1983) described a one-to-one 
evaluation process they used in a curriculum development project 
for the Kingdom of Saudi Arabia. Because of their project's needs 
and constraints, they selected a learner verification approach as 
the most appropriate. They found that the clinical procedure was 
efficient, cost-effective, and reliable. 

The clinical approach to learner verification involved work- 
ing with one student at a time. Each student, as he or she worked 
through a complete learning sequence, was closely observed by an 
experienced evaluator and subject matter expert. When a student 
encountered difficulty, the evaluator questioned the student until 
the source of difficulty yas identified. The subject-matter 
expert assisted in equipment setup and was responsible for student 
assessment on the learning objectives. Incorrect answers to test 
items were discussed with the student so that a determination 
could be made as to why the student chose the incorrect answer. 

The authors noted that at the end of the evaluation session, 
the evaluator debriefed the student by asking a series of ques- 
tions about the material and its effectiveness. Once the student 
responses and comments of the evaluator and subject-matter experts 
were summatized, revisions were made by the development team. A 
second one-to-one evaluation session was conducted. The authors 
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noted that additional evaluation sessions were not found to be 
cost -effective or time-effective. 



Small-Group Trial 

In the formative evaluation sequence proposed by Dick and 
Carey (1978) , small-group trials follow one-to-one evaluations. 
After the materials have been revised on the basis of the one-to- 
one evaluation, the typical procedure is to select 10-20 students 
to participate in a small-group evaluation of the revised mate- 
rials. Usually, the materials are administered in the manner in 
which they are expected to be used when they are in their final 
form. Pretests are administered before instruction begins. After 
the students complete the materials, they take a posttest. Ac- 
cording to Dick and Carey, the instructor does not intervene 
during the instructional process unless equipment or instructional 
problems prevent the student (s) from advancing through the mate- 
rials. The problems encountered by the students and how the 
problems were solved should be noted in the evaluation data. Dick 
and Carey noted that student attitude questionnaires and discus- 
sions with students are appropriate during the small group trial. 

Recent research found that it might be possible to eliminate 
either the one-to-one or small-group stage in the formative eval- 
uation sequence proposed by Dick and Carey (1978) and still main- 
tain effective revision of the instructional materials (Kandaswamy 
et al. 1976; Carey 1978; Wager 1983). Although these studies had 
limitations, their results indicated that formative evaluators 
might save time and financial resources by eliminating either the 
small-group step or the one-to-one step from their evaluation plan 
without sacrificing effectiveness. 



Local Trial and Fi eld Test 

Bastion et al. (1983) described a formative evaluation 
approach that combined the features of the one-to-one and small- 
group trials into one evaluation stage that they referred to as 
local trial. The second stage of their evaluation approach re- 
flected those procedures of a field test described by Dick and 
Carey (1978). 

During the local trial, a draft of the curriculum materials 
is submitted to a subject-matter expert who is not a member of the 
curriculum development team. The subject-matter expert reviews 
the materials for content and recommends revisions. In addition, 
a small-group trial is conducted with typical learners. Students 
are pretested, posttested, given an attitude questionnaire, and 
encouraged to write comments directly on the materials. 
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During the field-test stage, the curriculum materials are 
tested in a variety of settings under conditions similar to reg- 
ular course instruction. School-site staff and teachers admin- 
ister the curriculum materials, pretests and posttests, and cur- 
riculum evaluation questionnaires. The curriculum development 
team conducts interviews with school-site administrators, teach- 
ers, and students to obtain additional formative evaluation 
information. 



Four-Sta ge Formative Evalu^h ion 

Perhaps the most ideal formative evaluation approach for 
facilitating instructional material development, assessment, and 
revision is the approach described by Sanders and Cunningham 
(1973) and Sarapin (1961). Sarapin developed a symbolic analogue 
of the framework proposed by Sanders and Cunningham to simplify 
the essential elements of a comprehensive formative evaluation 
effort. This symbolic analogue, or conceptual model, is presented 
in figure 1. 



Identify & 
Order Goals 
1.0 



Evaluation 
of 
Goals 



Identify & 
Operationalize 
Objectives 

2.0 



Evaluation 
of 

Objectives 



Develop 
Interim 
Materials 



3.0 



Evaluation 
of 
Interim 
Materials 




Evaluation 
of 

Products 



Judgments, Revisions, and Feedback 



SOURCE Sarapin, M , 1981, p. 6. 



Figure 1. Conceptual model of four-stage formative evaluation framework 
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At the procedural level , Sarapin developed a formative eval- 
uation flowchart , shown in figure 2, that made the conceptual 
model operational. The flowchart, according to Sarapin, is a 
heuristic device that should be adopted as needed for each cur- 
riculum formative evaluation effort. A detailed description of 
each of the four stages of this model was provided by Sarapin 
(1981). 



enter 
formative 
sequence 




Figure 2- Flowchart of four-stage formative evaluation 



Instructional Quality Inventory 

Montague et al. (1983) noted that the Instructional Quality 
Inventory (IQI) is a formative evaluation process that was de- 
signed initially for military instructional development. The IQI 
is an internal review process that "can be applied to any sys- 
tematically developed program of instruction that has goals and 
test items and instruction tied to the goals" (Montague et al. 
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1983, p. 11). The IQI process, in essence, checks to see if test 
items are consistent with the instructional objectives. Once the 
objectives and test items are consistent with each other, the next 
step is to ensure that the instructional materials are consistent 
with the objectives and test items. 

nontague et al. (1983) presented an overview of ;he IQI 
procedures. Objectives and test items were classified by a task- 
content matrix. The IQI consisted of five procedures that judged 
the adequacy of objectives, test items, and instructional mater- 
ials. The IQI has been found to be a cost-effective process 
compared with other formative evaluation methods. The authors 
notedr however, "that the IQI does not address either adapting 
instruction to students or the sequenci j or structuring of con- 
tent" (p. 13). 



Respons ive Evaluation 

Stake (1975) h^s described the process of conducting a 
responsive evaluation. 

To do a responsive evaluation, the evaluacor conceives 
of a plan of observations and negotiations. He arranges 
fcr vaiious persons to observe the program, and with 
their help prepares brief narratives, portrayals, pro- 
duct displays, graphs, etc. He finds out what is of 
value to his audiences, and gathers expressions of 
worth from various individuals whose points of view 
differ. Of course, he checks the quality of his 
records: he gets program personnel to react to the 
accuracy of his portrayals; and audience members to 
react to the relevance of his findings. He does most 
of this informally — iterating and keeping a record of 
action and reaction. He chooses media accessible to his 
audiences to increase the likelihood and fidelity of 
communication. He might prepare a final written report, 
he might not — depending on what he and his clients have 
agreed on. (p. 14) 

Responsive evaluation makes extensive use of interviews (sometimes 
called conversations), observations, document reviews, and record 
reviews in and around a given program. The concerns and issues 
are gathered from the persons, records, and documents involved. 
In responsive evaluation, the evaluator determines how best to 
collect data. The human instrument is most frequently used, but 
other means are also employed to collect needed data. Case 
studies are frequently used in responsive evaluation situations 
because of the concern for a specific "case" and because they 
readily lend themselves to data collection via interviews, obser- 
vations, document review, and record review. 
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Summary 



ERIC 



A clcissif ication of ideal information needs for formative 
evaluation would show that no information should be overlooked, 
given consideration of project resources and constraints. The 
emphasis of formative evaluation was described as the process of 
collecting data on instructional materials for the purpose of 
revising the materials to itake them as effective as possible. The 
foundation for formative evaluation for curriculum development and 
improvement was the evaluation methodology reported by numerous 
scholars. Although numerous formative evaluation approaches have 
been developed, the following six approaches were identified as 
being pertinent to the Principles of Technology formative evalu- 
ation effort: (1) clinical approach, (2) small-group trial, 
(3) local trial and field test, (4) four-stage formative, 
(5) instructional quality inventory, and (6) responsive evalu- 
ation. A brief description of each approach was provided. An 
assessment of these six approaches, project constraints, essential 
features of the curriculum, and evaluation standards is included 
in the following chapter. 
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CHAPTER 3 



FORMATIVE EVALUATION FOR PRINCIFLES OF 
TECHNOLOGY CURRICULUM DEVELOPMENT EFFORT 



Principles of Technology Curriculum Overview 
Printed Teaching Materials 

The Principles of Technology student text is systematically 
divided into 14 units. Each unit covers one technical concept and 
contains an overview of the unit, subunits, and a summary. Each 
subunit deals with the unit's major technical concept as it ap- 
plies in one of the four energy systems, and each subunit has 
lecture materials, a suggested demonstration to use with the 
lecture portion, a math skills lab, and two hands-on physics 
application labs. A summary of the important concepts concludes 
the unit. A glossary of key words and activities for improving 
math skills are included in the unit's appendix. Finally, an end- 
of-unit exercise is provided to review the main ideas and defini- 
tions presented m the units. 

The target audience for units 1-7 and units 8-14 are 11th and 
12th graders, respectively. Typically, the 14 units are presented 
over 2 years and in the sequence delineated in figure 3. Further- 
more, 10th graders and students who are in the academic tracks may 



First Year Unit! 
Suggested Sequence 



Second Yaw Units 
Number Sequence 



Second Year Units 
Optional Sequence 



FORCE TRANSFORMERS 



RESISTANCE 



RATE 



WORK 



FORCE 



TIME CONSTRAINTS 
t 



OPTICAL SYSTEMS 



RADIATION 



TRANSDUCERS 



ENERGY CONVERTORS 

f 



WAVES AND VIBRATIONS 

i 



MOMENTUM 



TIME 
CONSTRAINTS 



SOURCE Center for Occupational Research and Development, 1985, p A 3 



OPTICAL 
SYSTEMS 



RADIATION 



V.AVFS AND 
VI? .ATIONS 



I MOMENTUM 



1ST YEAR 
UNITS 



TRANSDUCERS 






ENE 
CONVE 


RGY 
RTERS 



•In the second V—t, Momentum mutt he taught hnt 
The order of mitructton ■• then f'txibible with some 
eaceptiom Radiitton muil i exrfde Optical Syil#m, 
Wives and Vtbfitions must prteedf Radictton Energy 
Converton mutt Precede Trsmduceri 



Figure 3. Principles of technology instructional unit sequence 
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find the curriculum useful and beneficial. However, students 
enrolled in Principles of Technology should be able to read at 
least at the 8th-grade level and have satisfactorily completed 1 
year of high school general mathematics. Previous or concurrent 
enrollment in algebra is desirable. 

The teaching plan for a typical unit is depicted in figure 4. 
Each rectangle represents 50 minutes of instruction. Most units 
require 26 class sessions. The first class for each unit is an 
introduction and overview of the unit's content. The first two 
sessions of the subunit include the video presentation and lec- 
ture and discussions. The third session is a math skills lab. 
The next two sessions are hands-on physics application labs. The 
last class session of the subunit is a review of the material. 
The last class of the unit is a unit summary and test. 



UNIT 

OVERVIEW 
CLASS 1 



SUBUNIT 1 
CLASS 2-7 



CI 



C2 



M 



LI 



L2 



MECHANICAL 
SYSTEMS 



SUBUNIT 4 
CLASS 20-25 



SUBUNIT 2 ( 
CLASS 8-13 I CI 



SUBUNIT 3 
CLASS 14-19 



CI 



CI 



C2 



C2 



C2 



M 



M 



M 



LI 



LI 



LI 



L2 



L2 



L2 



FLUID 
SYSTEMS 



ELECTRICAL 
SYSTEMS 



THERMAL 
SYSTEMS 



UNIT 

SUMMARY 



Figure 4. Schematic of a typical instructional unit teaching plan 



For eacn instructional unit there is a teacher's guide, which 
provides suggestions for teaching each class session. The guide 
is not a set of rigid rules or a substitute for teacher ingenuity, 
but it is a useful tool for implementing the Principles of Tech- 
nology course. 
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Video 



r«o There are about 78 video programs that total approximately 
500 minutes infused throughout units 1-14. The video segments 
introduce ideas presented in the text. Through the video program 
students are taken to the workplace settings where technicians are 
employed. The video segments were designed as a tool for putting 
variety into the classroom teaching pattern. 



Equipment and Facilii-iP S 

A standard-size classroom or laboratory space sufficient to 
accommodate one to five laboratory stations per class is needed to 
implement the Principles of Technology units. Depending upon 
class size and availability cl equipment, lab space must be flex- 
ible to accommodate a variety of lab station activities. A video- 
cassette player (1/2" Beta, 1" VHS, or 3/4" U-matic) and a color 
monitor are needed for the video segments. The cost of equipping 
each lab station for the first year is approximately $4,000- 
$6,000. Demonstration and laboratory equipment may be locally 
available, may need to be fabricated, or may need to be ordered 
from commercial vendors. Laboratory and equipment specifications 
are delineated in the teacher's guide. 



First-Year Formative Ev aluation r.ffort 

The National Center designated study Design and Assessment of 
the Formative Evaluation of Principles of Technologv Curriculum 
Materials had a 12-month contract that commenced on' January 16, 
1985. Thus, National Center project staff entered the Principles 
of Technology 2-year formative evaluation effort at approximately 
the midpoint of the AIT evaluation time line. In fact, the evalu- 
ation design for units 1-7 was developed and implemented prior to 
the entry of the National Center into the formative evaluation 
effort. 

The first-year evaluation plan, as developed by AIT, is 
depicted in table 2. The entire data collection instrument pack- 
age used by AIT is presented in appendix B. 

Descriptive statistics were used to analyze the data from the 
student and teacher questionnaires. Respon.es from '■pen-ended 
questions were summarized and evaluated by AIT project staff as to 
their merit for improving the curriculum materials. Student 
pretest and posttest scores were subjected to analysis of covari- 
ance that controlled the effects of student sex and grade level, 
teacher background, and teaching pattern. 



KssenHal Featu res of 
Principles of Technology Curriculum Effort 



One of the initial steps in the design of the formative 
evaluation by National Center staff was the identification of 
those features considered essential and unique to the Principles 
of Technology curriculum. The essential features, together with 
the likely constraints and the evaluation standards established by 
the Joint Committee on Standards for Education Evaluation (util- 
ity, feasibility, propriety, and accuracy), served to guide the 
task staff in analyzing alternative formative evaluation designs. 
Selecting suggestions for strengthening the formative evaluation 
by National Center staff was essentially a critical judgmental 
process involving trade-offs among the essential features and 
evaluation standards. The essential features of the Principles of 
Technology curriculum effort, identified by National Center staff, 
are discussed in the following sections. 



TABLE 2 



PRINCIPLES OF TECHNOLOGY EVALUATION PLAN 



Pro J ect 
Materials 


Data Source 


Assessment Focus 


Instrument 


When 


Video/Student 
Handbook/Lab 8 


Students 


Overall Instruc- 
tlonel Impact 


Pre/po attest 
Queetionnei re 


1 subunit 
for each 
unit and 
1 unit 


All 


Teachers 


Manageability of 
component8 v clarity 
of material 8 f appro- 
priateness for tar- 
get audience* per- 
ceived aducaclonal 
worth 


Questionnaire 
and structured 
log 


At con- 
clusion 
of each 
unit 


All 


Oonsortl um/Oon- 
sortlum Review 
Tea* 


Clarity, appropH- 
atenese, meterlels 
link 


None for re- 
view team/ 
Quest1onna1 res 
for full con- 
sortium 


As eech 
unit 18 
developed 
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Instructional 



The comprehensive student text, video, and teacher's guide 
are essential for the successful implementation of the Principles 
of Technology curriculum. Therefore, these materials form a total 
instructional package that needs to be included in the iormative 
evaluation. For example, do the video and printed materials 
correlate in terms of textual and graphic illustration? Is the 
terminology consistent throughout the instructional materials? 



Equipment and Fac ilities 

The application laboratories are essential for successful 
implementation of the curriculum. Lab space must be flexible to 
accommodate a variety of lab station activities. A videocassette 
player and a color monitor are required. Specifically, was con- 
sideration given to equipment cost, utility, adaptability, and 
availability? Are laboratory and equipment setup specifications 
provided? If special equipment needs to oe fabricated, are design 
specifications adequately described? 



Teacher Background 

Teachers from the existing faculty who have an interest in 
vocational students should teach Principles of Technology. These 
teachers should be familiar wit s or willing and able to become 
familiar with the physics and mathematics in the course. Team 
teaching may be required if qualified teachers are not available. 



Student Background 

The Principles of Technology curriculum is designed for 11th- 
and 12th-grade students who are interested in technical careers. 
Tenth-grade and nonvocational students also may want to enroll, 
and they should be accommodated. Students should have at least an 
eighth-grade reading level and should have had at least 1 year of 
general mathematics. Previous or concur re: t enrollment in algebra 
would be helpful. 

Although the following features are not unique to the cur- 
riculum, they are essential for successful implementation. 



School Counselor Coo peration 

SchooJ counselors should be knowledgeable of the purpose, 
content, and prerequisites of the Principles of Technology 
curriculum so that proper counseling can be given to potential 
students and to currently enrolled students. 
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School Administrator Support 

School-site administrative support and leadership are essen- 
tial to the successful implementation of Principles of Technology. 
School-site administrators play a pivotal role in eliciting the 
support of the school board, the community, and the school's 
faculty and staff. 



Community Support 

Parents, business and industry, and concerned citizens should 
' e knowledgeable of the purpose and general content of the Princi- 
ples of Technology curriculum so that meaningful community cooper- 
ation can be elicited in the implementation effort. 



School Characteristics 

Individual school culture and donate would affect implemen- 
tation of the Principles of Technology curriculum, as would the 
size of the school, students' and teachers' backgrounds and their 
relationships with each other, their attitudes toward new curricu- 
lum and toward schooling in general, school-site leadership, and 
the socioeconomic status of the student population. 



Information Activities 

A well-planned public relations effort is essential for 
eliciting school and community support for the Principles of Tech- 
nology curriculum. Information activities are needed at the 
National, State, and local levels to inform potential students, 
counselors, teachers, school administrators, and community and 
business leaders about the new curriculum. 



Constraints on Formative Ev aluation Design Potential 

Numerous constraints on the Principles of Technology forma- 
tive evaluation effort were identified by National Center and AIT 
staff. Although these constraints did not make the evaluation 
impossible, they did alert the evaluators to potential problems 
and limitations that needed to be addressed in the formative 
evaluation design. The following sections highlight the more 
serious constraints on the Principles of Technology formative 
evaluation. 
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Insuffic ient Resnnrres 



Two obvious and uncontrollable factors that limit the Prin- 
ciples of Technology formative evaluation effort are time and 
financial resource constraints. Given pilot-test time lines of 
September 1984 through June 1986 for units 1-7 and September 
1985 through June 3 986 for units 8-14, time became a severe re- 
striction on the formative evaluation effort. Moreover, National 
Center staff were limited by time in what they could propose as a 
useful and feasible formative evaluation supplemental design, and 
their project budget constraints precluded them from conducting 
evaluation activities that they proposed. Likewise, the AIT 
project staff members were limited by their budget on the number 
and type of additional evaluation activities they could undertake. 
In summation, given time and budget constraints, the development 
of an effective, efficient, and feasible formative evaluation 
design that would enhance current Principles of Technology evalu- 
ation activities presented a challenge to all involved. 



Data Collection Limitations 

The rights and welfare of human subjects participating in the 
formative evaluation needed to be respected and protected. Moral, 
ethical, and legal codes needed to be followed. To ensure that 
information was not obtained illegally or unethically, data col- 
lection efforts liai to conform to the rights of human subjects. 
This reality imposed limitations on "what" and "how" information 
was collected. 



External Constrains 

A number of persistent external constraints affected the 
project. 

Equipment problems. Test site variability in the success 
c r caining correct laboratory equipment and proper equipment 
semp was an uncontrollable factor that affected test site evalua- 
tion results. 

Lack of demographic contex tual data , student and ceacher 
background data may not be representative of the target popula- 
tion. Information about school climate and culture and about 
students' and teachers' perceptions of the curriculum and class- 
room climate was lacking. The lack of demographic/contextual data 
imposed restrictions on interpretation of evaluation results. 



ERIC 



19 

28 



variability in teaching pattern . Variability in length of 
classroom instruction, nu-nbor of classes per unit, number and type 
of homework assignments, teacher enthusiasm, and out of class 
assistance constituted uncontrollable factors that affected evalu- 
ation results. 



First-Year Evalua tion Design 

In essence, che first-year evaluation design was a field test 
that utilized a one-group pretes. -posttest design as a means of 
measuring criteria performance. Threats to the internal validity 
of this design were numerous. No control groups were utilized; 
therefore, the following threats to validity imposed serious 
constraints on the interpretation of posttest results. 

History . For examp:*, during the time of a particular unit 
of instruction, did students receive any outside learning assist- 
ance from relatives, tutors, TV, friends, and so forth? If so, it 
would be difficult to interpret posttest scores accurately. Did 
the instructional unit account for the difference between pre- and 
posttest scores, or did the outside help account for the differ- 
ence? 

Testing . Since the pre- and posttest instruments were iden- 
tical, there may be a practice effect or pretest sensitization. 
However, this threat was addressed by minimizing memory effects. 

instrumentation . Perhaps the most serious threat to the 
validity of the results pertained to the validity and reliability 
of the pre- and posttests. Interpretation of the posttest results 
must consider the lack of validity and reliability informacion for 
the measurement instruments. 

Regression . This is a subtle phenomenon that needs to be 
considered when interpreting posttest results. Regression sug- 
gests that students who scored low on the pretest will tend to 
regress toward the mean on the posttest. Conversely, students who 
scored high on the pretest will tend to regress toward the mean on 
the posttest. 

Selection . Neither students nor classrooms were randomly 
selected. Test site selections appeared to have been volunteered 
by respective State staff members. In addition to threats to 
internal validity, thore were external validity threats to the 
one-group preteat-posttest design. 

Hawthorn e effect . Since the students knew that they were 
participants in an evaluation, they may have tried harder than 
they normally would have to learn the material. Thus, their 
posttest scores may have increased as a result of this special 
attention. 

20 



ERIC 



30 



Novelty effect. The newness of the curricula may excite 
student interest. This effect is similar to the Hawthorne effect. 



NO control group. A lack of a usable comparison group re- 
stricted interpretation of posttest results. Perhaps the same 
students in a technical physics class taught by a physics teacher 
would obtain the same or better results on the posttest. 

Lack of random assignment. Students or teachers were not 
randomly assigned to treatments. The generalization of results to 
students and teachers who did not participate in the evaluation is 
not valid. 

Although the aforementioned threats to internal and external 
validity pertain to behavioral research in general and not to 
formative evaluation pilot testing per se, these threats should be 
considered when interpreting unit posttest scores, in addition, 
it is impossible or impractical to deal with many of these prob- 
lems in the educational setting. 



Evaluation Standards 

The evaluation standards established by the Joint Committee 
on Standards for Educational Evaluation (1981) served to guide 
National Center task staff in analyzing alternative formative 
evaluation designs. The Joint Committee identified four important 
attributes of an evaluation: utility, feasibility, propriety, and 
accuracy, and stated, "The Committee is satisfied that standards 
which shape an evaluation so that it has these four characteris- 
tics are necessary and sufficient for sound evaluation in educa- 
tion" (p. 13). The four standards, as they applied to formative 
evaluation, are described in the following subsections. 



Utility 

Information scope and se lection . The information collected 
by the formative evaluation should be sufficient to support a 
judgment of worth and merit and should answer all relevant forma- 
tive evaluation questions. 

Clarity of information. The information collected should 
respond clearly and without generalities to the formative eval- 
uation objectives, provide a firm foundation for formative evalu- 
ation conclusions and recommendations, and be characterized by 
conciseness and logical development so that it communicated to the 
formative evaluation audiences. 
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Timeliness, The formative evaluation should provide infor- 
mation to the curriculum developers at the time when the infor- 
mation can best be used. The most critical information needs of 
the curriculum developer must be met on time to avoid delays in 
curriculum revision decisions. 



Feasibility 

Practical procedures . The formative evaluation procedures 
suggested by National Center staff should be realistic and prac- 
tical, given the time and financial constraints inherent in the 
formative evaluation project. School and classroom disruptions 
should be kept to a minimum. 

Political viability . The formative evaluation procedures 
should be politically viable given the various interest groups and 
stakeholders in the Principles of Technology curriculum project. 
In essence, the formative evaluation procedures are politically 
viable if their purpose is achieved despite pressures created by 
the formal and informal organizational power structures of the 
State and local levels. 

Cost-effectiveness . The evaluation has to produce informa- 
tion of sufficient value to justify its expense. The question of 
whether there are alternative evaluation approaches that would 
produce more useful information at the same or less cost needs to 
be addressed. The information needed should be obtained as eco- 
nomically as possible. 



Propriety 

Rights of human subjects . The formative evaluation should 
protect the rights and welfare of human subjects. Legal, ethical, 
and common sense considerations should be followed. 

Conflicts of interest . The formative evaluation should take 
into consideration the fact that conflicts of interest may arise 
during the formative evaluation effort. Strategies should be 
developed for dealing with conflicts so that the evaluation pro- 
cess and results will not be compromised. 

Balanced reporting. The formative evaluation should be 
complete and fair in its presentation of strengths and weaknesses, 
and both negative and positive aspects of the curriculum evalu- 
ation should be reported. Strengths and weaknesses should not be 
manipulated to please partisan interest groups. All evaluation 
findings, even those that might prove embarrassing to partisan 
interest groups, should be included in pilot-test findings. 
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Accuracy 



Clearly identified nhjecr. The object of the formative 
evaluation should be clearly identified and described realis- 
tically. Unique features of the object should be identified. In 
summation, the descriptions and unique features of the object 
should be a valid characterization of the object. 

Appropriate information spumes. The formative evaluation 
must provide adequate information to the curriculum developers. 
Multiple information sources should be utilized. Information 
sources should be tapped using a variety of methods, such as 
interviews, surveys, observations, and document reviews, in 
essence, the information sources should answer the evaluative 
questions. 

Adequacy of contextual in formation . Contextual information 
snould be used in the interpretation of formative evaluation 
results. For example, the curriculum developers should know 
whether a particular unit's success or failure was influenced by 
student socioeconomic status, teacher background, student academic 
background, school climate, teacher and administrator support or 
resistance toward the curriculum, and/or community support or 
apathy toward the curriculum. 

Purposes and procedures explicated . Purposes and procedures 
of the formative evaluation must be understood by those involved 
in the formative evaluation effort. Specifically, the objectives 
of the evaluation need to be clear. Procedures on how information 
was collected, organized, analyzed, and reported should be de- 
scribed in sufficient detail so that other evaluators could 
conduct a similar evaluation in other settings. 

Propriety rights of human subjects . The rights and welfare 
of human subjects need to be respected and protected. Legal, 
ethical, and common sense considerations need to be followed. 

Conflicts of interest-. Conflicts of interest should be 
avoided. If conflicts of interest arise, they should be dealt 
with so that the evaluation process and results are not com- 
promised. 

Balanced reporting. The formative evaluation should be 
complete and fair in its presentation of strengths and weaknesses. 
Both negative and positive aspects of the curriculum evaluation 
should be reported. Strengths and weaknesses should not be 
manipulated to please partisan interest groups, in fact, evalua- 
tion findings that might prove embarrassing to partisan interest 
groups should not be omitted from evaluation findings. 
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As sessment of Alternative Formative Evalu ation Approaches 



The essential features of the Principles of Technology cur- 
riculum, together with the constraints and the evaluation stand- 
ards established by the Joint Commission, served as a guide for 
assessing the formative evalua'.ion approaches identified by 
National Center staff, n essence, the assessment was a critical 
judgment process that involved trade-offs among the essential 
features, information needs, constraints, and evaluation stan- 
dards. Suggestions for strengthening the first-year Principles of 
Technology formative evaluation effort emerged from this process. 

The results of the assessment of information needs in rela- 
tion to evaluation standards are delineated in table 3. The 
information needs specified in table 3 are identical to those 
listed in table 1. Ideally, the Principles of Technology forma- 
tive evaluation design should satisfy the information needs list- 
ed. However, project constraints limited the number and type of 
data collection activities that could be conducted. 



TABLE 3 

ASSESSMENT OF INFORMATION NEEDS IN RELATION TO 
EVALUATION STANDARDS 



Evaluation Standards 



Evaluation Design 
Information Needs 


Util- 
ity 


Feasi- 
bility 


Pro- 
priety 


Accu- 
racy 


I. Internal information 
A. Descriptive 

1. Physical speci- 
fications 

2. Rationale, goals, 
objectives 

3 Content 


Y 


Y 


Y 


Y 


B. Critical Appraisal 

1. Developer 

2. Subject matter 
experts 

3. Students using the 
material 

4. Teachers using the 
materials 


Y 


Y 


Y 


Y 
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TABLE 3 — Continued 



Evaluation Design 
Information Needs 



Evaluation Standards 



Util- 
ity 



Feasi- 
bility 



Pro- 
priety 



Accu- 
racy 



II. External Information 

A, Interim Effects of thu 
Materials on Student 
Behavior 

1. Achievement 

2. Attitude 

3. Skill 

4. Interest 

5. Commitment 



B. Interim Effects of the 
Materials on Teacher 
Behavior 

1. Attitude 

2. Interest 

3. Commitment 

4 . Competency 

5. The community 



III. Contextual Information 

A. Student Character- 

istics 

B. Teacher Character- 

istics 

C. School Character- 

istics 

D. Community Character- 

istics 

E. Curricular Character- 

istics 



KEY: Y= Yes, satisfactorily meets the criteria 
P= Partially meets the criteria 
N= No, does not meet the criteria 



Specifically, it was judged feasible to investigate the 
interim effects of the curriculum materials on students and teach- 
ers not using the curriculum. Project constraints also prevented 
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an in-depth collection of contextual information about student, 
teacher, school, and community characteristics. Project staff 
judged information collection activities concerning the interim 
effects of the curriculum materials on student and teacher be- 
havior to be feasible. Specifically, the measurement of student 
and teacher attitudes toward the curriculum as well as the mea- 
surement of student achievement appeared to be the most feasible 
data collection activities on interim effects. Internal informa- 
tion needs were, for the most part, being met by existing Prin- 
ciples of Technology formative evaluation activities. 

The second stop in the critical judgment process was to 
assess the alternative formative evaluation approaches against the 
evaluation standards criteria. The information needs that were 
judged as meeting the evaluation standards in the first assessment 
were included under the utility standards of the second assess- 
ment. The four standards were then crossed with the alternative 
evaluation approaches. National Center staff assessed these 
alternative approaches by assigning a numerical value to each cri- 
terion. The results of this assessment are presented in table 4. 

The results of the alternative evaluation approach assessment 
revealed that no single approach emerged as the optimal method for 
the formative evaluation of the Principles of Technology curricu- 
lum. It was decided that an eclectic approach would most likely 
incorporate appropriate features of the approaches that best met 
the evaluation criteria. 

As shown in table 4, two approaches were judged to meet the 
evaluation criteria best: (1) local trial and field test and 
(2) responsive evaluation. The local trial and field test ap- 
proach was judqed more feasible than responsive evaluation. 
Conversely, responsive evaluation was judged more useful than the 
local trial and field test approach. The logical conclusion was 
to combine those features of both approaches that best fit the 
needs of the Principles of Technology formative evaluation effort. 
Consequently, the last step in the critical judgment process was 
to compare the first-year evaluation effort approach with features 
of the local trial and field test and responsive evaluation 
approaches so that elements of the eclectic approach could be 
identified that would strengthen current evaluation efforts. 



Selected Design Elements 
fcfl Strengthen Current Evaluation Efforts 

When the first-year evaluation approach was compared with the 
local trial and field test and responsive evaluation approaches, 
information voids were identified that could be filled in the 
second-year evaluation effort. Specifically, responsive evalu- 
ation could collect, through the case-study method, qualitative 
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TABLE 4 



ASSESSMENT OF ALTERNATIVE FORMATIVE EVALUATION 
APPROACHES AGAINST EVALUATION STANDARDS 





Evaluation Standards 




Utility 


Feasibility 


Propriety 


Accuracy 




Alternative Fomatlvs 


inioni t ion 


rrBCil Cfll r rO 


nights or 


Def anelble 


Total 


Evaluation AnnrnnrhAft 


neeca 


cedures Given 


Hunan Sub"* 


Information 


Points 






ijonetraints 


J acta 


Source 








and Essential 












PBB tUrOB 








CUnlcel Approach 












(One-to-One) 


1 


1 


F 


3 


10 


Smell-Group Trial 


3 


3 


5 


3 


14 


Local Trial and 












Field Test 


3 


5 


5 


5 


1B 


Four-Stage Formstlvs 


5 


1 


5 


5 


16 


Instructional Quality 












Invento ry 


1 


1 


K 

** 


5 


12 


Responsive Evaluation 


5 


3 


5 


5 


18 



KEY: 5= Satisfactorily net the criteria 
3= Partially met the criteria 
1= Minimally net the criteria 



contextual information that would be useful in helping to inter- 
pret the interim effects of the curriculum on students, teachers, 
and school-site staff. Interviews with teachers, students, and 
school administrators, in particular, would enhance the second- 
year evaluation effort. Moreover, classroom observations and 
document reviews would provide additional insights to the interim 
effects of the curriculum. 

In essence, the first-year evaluation was similar to a local 
trial and field test approach. Instructional units were submitted 
to subject-matter experts for content review. Although small- 
group trials were not conducted, the instructional units were 
field tested in a variety of settings (comprehensive high schools, 
vocational schools, teachers with varying academic and work ex- 
periences, students with varying academic and work backgrounds, 
and so forth). Students were pretested before each unit and were 



27 



ERIC 



37 



administered a posttest and attitude questionnaire at the comple- 
tion of each ur.it. For each unit, teachers completed a detailed 
forced choice and open-ended questionnaire that allowed them to 
record their daily reactions and suggestions for improving the 
unit. 

National Center staff believed that the student and teacher 
questionnaires could be strengthened by focusing their data col- 
lection activities on the essential features of the curriculum. A 
Likert-type format was proposed. The data collection approach 
proposed by National Center staff for strengthening the Principles 
of Technology formative evaluation is presented in appendix D. 
The data collection package was comprised of four major compon- 
ents: (1) a student reactionnaire; (2) a teacher reactionnaire; 
(3) a teacher unit daily log; and (4) case study guidelines for 
student, teacher, and administrator interviews, classroom obser- 
vations, and document and record reviews. The total package was 
designed to fill the information voids of the first-year evalu- 
ation effort, given the project constraints and evaluation cri- 
teria. 



28 

38 



CHAPTER 4 



ASSESSMENT OF IMPLEMENTATION OP ESSENTIAL 
EVALUATION DESIGN FEATURES 



Assessme nt Design 

National Center staff assessed the extent to which the 
curriculum evaluators adhered to the National Center staff's 
evaluation suggestions that were to be implemented in the second- 
yea.: field-test of the Principles of Technology curriculum. The 
assessment was guided by the four evaluation standards described 
by the Joint Committee on Standards for Educational Evaluation 
(1981). In essence, the assessment framework formed a matrix with 
the evaluation standards on one axis and the essential evaluation 
design features on the other axis. The assessment framework is 
depicted in figure 5. A description of each assessment topic area 
is presented in the following sections. 

Utility 

Information scope and se lection . The audit should assess the 
extent to which the formative evaluation addressed the information 
needs of the curriculum developers. Was the information collected 
sufficient to support a judgment of worth and merit? Was the 
information useful? Were all relevant formative evaluation ques- 
tions answered by the information collected from the data collec- 
tion activities? 

Clarity of information. The audit should assess the extent 
to which the formative evaluation information collected was under- 
standable to the curriculum developers. Did the information 
respond clearly and without generalities to the formative evalua- 
tion objectives? Did the information provide a firm foundation 
for formative evaluation conclusions and recommendations? Was the 
information characterized by conciseness and logical development? 
In sum, did the information communicate to the formative evalua- 
tion audiences? 

Timeliness. The audit should assess the extent to which the 
formative evaluation information was provided to the curriculum 
developers at the time when the information could best be used. 
Was the formative evaluation information received by the curric- 
ulum developers on scheduled time line dates? Were any curriculum 
revision decisions delayed as a result of late formative evalua- 
tion information? In sum, were the curriculum developers' most 
critical information needs met on time? 
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Feasibility 



Practical procedures. Audit interviews and classroom obser- 
vations should assess the extent to which formative evaluation 
procedures suggested by National Center staff were realistic given 
the time and financial constraints inherent in the formative 
evaluation project. Specifically, were the National Center evalu- 
ation p-ocedures a practical approach to the formative eval> 'cion 
of the Principles of Technology curriculum? Did these procedures 
minimize school and classroom disruptions? Moreover, was the 
evaluation conducted on a daily basis without the presence of 
curriculum evaluators? Were teachers informed of the purpose an 
procedure of the evaluation effort? Were teachers instructed oi 
how to implement t.ie instructional units? Did the teachers fee' 
that they were well-prepared to teach the instructional units? 
Were the students informed of the evaluation purpose and proce- 
dures? Did the students feel that they were adequately prepared 
for participating in the formative evaluation? 

Political viaLUitr. Audit interviews and classroom 
observations should assess the extent to which the formative 
evaluation procedures were politically viable given the various 
interest groups and stakeholders in the Principles of Technology 
curriculum project. In essence, thr formative ev luation 
procedures were politically viable if the purpose of the 
evaluation was achieved despite pressures created by the formal 
and informal organizational power structures of the State and 
local levels. Specifically, did any political conflicts erupt 
over the evaluation effort? so, what impact did these 
conflicts have on achieving t. . purpose of the evaluation? 

. . Cost-effectiveness. The audit should assess the extent to 
which the evaluation was cost-effective. The evaluation should 
produce information ot sufficient value to justify the expense of 
the evaluation. Were there alternative evaluation approaches 
that would produce more useful information at the same or less 
cost? Was the evaluation begun without a commitment of sufficient 
resources to ensure completion? For the information needed, was 
the evaluation conducted as economically as possible? 



Propriety 

R i ghts Of human sub j ects . The audit should assess the extent 
to which the rights and welfare of human subjects were respected 
and protected. Were legal, ethical, and common sense considera- 
tions followed? 

Conf l icts . o f i nterest . The audit should assess the extent to 
wh."-h conflicts of interest in the formative evaliation effort 
were avoided. If conflicts of interest arose, how were they dealt 
with so that the evaluation process and results were not compro- 
mised? * 



31 

41 



Balanced reporting . The audit should assess the extent to 
which the formative evaluation was complete and fair in its 
presentation of strengths and weaknesses. Were both negative and 
positive aspects of the curriculum evaluation reported? Were any 
strengths and weaknesses manipulated to please partisan interest 
groups? Were any evaluation findings that might prove embarrass- 
ing to partis^i interest groups omitted from pilot test findings? 



Accuracy 

Clearly identified object . The audit should assess the 
extent to which the object of the formative evaluation was clearly 
identified. Given project constraints, was the object of the 
evaluation described realistically? Were unique features of the 
object identified? Were the descriptions and unique features of 
the object valid characterizations of the object? 

Appropriate information sources . The audit should assess the 
extent to which the formative evaluation provided adequate infor- 
mation to the curriculum developers. Were multiple information 
sources utilized? Were information sources tapped with a variety 
of methods, such as interviews, surveys, observations, and docu- 
ment reviews? In essence, to what extent did the information 
sources answer the evaluative questions? 

Adequacy of contextual information . The audit should assess 
the extent to which contextual information was used in the inter- 
pretation of formative evaluation results. For example, the 
Principles of Technology curriculum developers should know whether 
a particular unit's success or failure was influenced by the 
students 1 socioeconomic status, teachers' background, students' 
academic background, school climate, teacher and administrator 
support or resistance toward the curriculum, and/or community 
support cj apathy toward the curriculum. In sum, what contextual 
factors were examined , to what extent were these factors used in 
describing the contextual conditions of the formative evaluation 
effort, and how adequate was the information in helping to inter- 
pret the outcomes of the formative evaluation? 

Purpospg and procedures explicated . The audit should assess 
the extent to which the purposes and procedures of the formative 
evalu< ;ion were understood by those involved in the formative 
evaluation effort. Were the objectives of the evaluation clear? 
Were procedures on how information was collected, organized, 
analyzed, and reported clearly delineated? If the evaluation were 
tc be replicated, are the purposes and procedures described in 
sufficient detail so that other evaluators could conduct a similar 
evaluation in other settings? 
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Furthermore, an overall assessment of the extent to which the 
essential features of the evaluation design were adhered to needed 
tc be addressed. Tn essence, the overall assessment served as a 
summative evaluation of the information obtained from the discrete 
components of the design depicted in figure 4. Specifically, were 
th< multiple data sources and data collection activities utilized? 
To what extent were the data collection procedures followed? Did 
all the field-test sites participate in the data collection 
activities? in sum, to what extent did the curriculum evaluators 
adhere to the essential evaluation features? 



Data Coll err ip n 

The data collection for the assessment of the evaluation 
suggestions proposed by the National Center was done through the 
review of appropriate information available at AIT headquarters, 
interviewing AIT evaluation staff members, observing Principles of 
Technology classes at two local PT pilot sites, interviewing a 
state-level liaison person, and interviewing students, administra- 
tors, and teachers in two local PT pilot sites. A copy of the 
topics/questions used to guide the interviews is provided in 
append. • F. in general, the interviews were unstructured, with 
t> to^cal areas and questions serving as a guide to ensure that 
ct ,ain topics were covered in the interviews, observations, and 
document or record reviews. 

The timing of this task (initiated after AIT/CORD had 
designed and implemented the formative evaluation) and the fact 
that Principles of Technology pilot teachers were generally behind 
schedule in implementing units 8-14 meant that it was almost 
impossible to secure meaningful feedback concerning National 
Center suggestions for changes in the formative evaluation effort. 
The case studies being conducted by the AIT staff were in the 
pilot stage at the time data for this report had to be collected. 



Assessment Summary 

After sharing the proposed second-year formative evaluation 
approach with AIT project staff, AIT responded with a revised 
version of the data collection instruments (appendix E). The 
specific changes made by the AIT project staff are noted as fol- 
lows. 
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o Student R eactionnaire 

— Unit Objectives - Items 1, 2, and 3 were reworded, 

— Unit Content - Items 5, 7, and 11 were deleted, 

— Unit Posttest - Entire domain deleted, 

— Unit Video - Items 16, 18, and 19 were deleted. Two 
items we ce added. 

— Unit Text - No changes, 

— Laboratory Activities - Items 26 and 27 deleted, 

— Unit Difficulty - Item 30 deleted. Item 33 reworded, 

— Student Satisfaction - Items 36 and 37 deleted. Item 38 
reworded. 

o Teacher Reactionnaire 

— Unit Objectives - Items 2, 4, and 5 deleted. 

— Unit Instructional Activities - Items 12 and 13 deleted. 

— Unit Content - Items 14, 16, 17, and 18 deleted. One 
item added. 

— Unit Posttest - Entire domain deleted. 

— Perceptions of Instructional Planning - Item 35 deleted. 

Furthermore, the student and teacher reacticnnaires were 
renamed as student and teacher questionnaires. After each domain 
item set, a general commencs statement was added. No substantive 
changes were made to the unit daily log and in the case study 
guidelines. 

Case Studies 

Information about the implementation of th case studies was 
based on a pilot test of the case study procedu 3s by the AIT 
evaluation staff at a Principles of Technology site in a large 
local comprehensive secondary school. 

Utility , The AIT evaluation staff considered the information 
collected through interviews, classroom observations, and docu- 
ment/record reviews to be very useful. The information collected 
added depth of understanding to that collected from the question- 
naires. When data from the questionnaires and the case studies 



were combined, the data were generally more understandable and 
more logical. Considerable data were generated in the pilot case 
study, if the information is to be utilized optimally by 
curriculum developers, it will need to be reduced to a concise 
format that communicates significant points easily. It may be 
necessary to concentrate the case study effort in the sarly part 
of the curriculum field test so that feedback can be provided to 
curriculum developers in a timely manner. 

Feasibility. Time and financial constraints are inherent 
difficulties in conducting case studies. National Center staff 
had suggested six case studies be conducted. Within the limits of 
time and finances, the AIT staff developed plans to complete four 
case studies. The time and cost of conducting the case studies 
was judged to be worthwhile, considering the high value placed on 
the information produced by the case studies. The case studies 
were particularly valuable in that they permitted evaluators to 
probe various interest groups and stakeholders who did not receive 
questionnaires. 

Propriety. All sites and individuals involved in case stu- 
dies were assured anonymity. No legal or ethical concerns were 
expressed by case study participants. The individuals conducting 
the case studies stressed the nature of formative evaluation, 
thereby lessening the potential for conflicts arising regarding 
the evaluation activities. The case studies provided an excellent 
opportunity to present strengths and weaknesses of the PT curri- 
culum. Furthermore, the case studies allowed the evaluators the 
flexibility of being able to triangulate information on site. 

Accuracy. At the time this report was written, none of the 
case study reports had been completed. Based on National Center 
staff knowledge of the PT effort and case study methodology, it 
appeared that the material to be developed for the case study 
reports reflected the reality of the situation in the local 
sites. 

Developer implementation. The developer (AIT) p\ans to 
conduct four case studies. National Center staff has recommended 
six case studies because of the enormous variation in teacher 
background, student recruitment, availability of laboratory equip- 
ment, and local or State support. Time and financial constraints 
prevented the developer from conducting six case studies. 

National Center staff considered it of vital importance for 
all of the individuals planning to conduct the case studies to be 
involved in a pilot case study so that the staff could reach 
consensus on topical areas to be studied and the emphasis to be 
placed on various aspects of the study, and, most importantly, 
could become more aware of the interpretations they place on 
various pieces of data. Two AIT staff who were to be responsible 
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for conducting the case studies participated in a 5-day pilot of 
the case study procedures. The pilot effort enabled the two AIT 
staff members to reach agreement on most procedures and techniques 
to be used in succeeding case studies. Additional details were 
finalized through the use of te". »ohone and mail communications. 



Unit Daily Log, Teacher Questionnaire . 
and Studen£_Ougstionnaire 

Very few of the PT pilot sites completed all seven units 
scheduled for the first year. This meant that the first part of 
the second-year program, which should have been devoted to units 
8-14 , was devoted to units 6 and 7 in most situations, in order 
to maintain continuity, the same evaluation instruments used for 
units 1-7 during the first year of the PT pilot were used for 
units 6 and 7 that were taught during the second year of the PT 
pilot. The changes proposed by the National Center staff and 
adopted by AIT were to be used in the formative evaluation of 
units 8-14. 

Because of the delays in starting unit 8 in most schools, 
there was minimal information available concerning implementation 
of the revised instruments at the time this report was written. 
The following information was obtained from AIT evaluation staff 
and local school site personnel who had completed unit 8. 

utility . The curriculum developers were generally pleased 
with the usefulness of information they received from the instru- 
ments. There may be some redundant information, but it was deemed 
important to have confirming information from students and teach- 
ers. The information was provided in a timely manner. Ques- 
tionnaires completed by students and teachers were returned to the 
curriculum developers upon the completion of a PT unit. The 
timing difficulties appeared to be more a case of teachers not 
progressing through the units as rapidly as anticipated, rather 
than any difficulties associated with the formative evaluation of 
the curriculum. 

Feasibility. National Center staff proposed several changes 
in the student questionnaire, teacher questionnaire, and the unit 
daily log. Earlier in this section, a summary was provided indi- 
cating AIT staff disposition of these suggestions. The changes in 
instrumentation were not perceived by teachers or students to be 
excessive in terms of classroom disruption. Teachers did indicate 
that completing the instruments was "a pain," but they understood 
and appreciated the need for such information. Teachers, adminis- 
trators, and students were unaware of any problems caused by the 
evaluation. National Center staff did propose adding to the 
instruments a limited number of items dealing with teacher and 
instructional effectiveness as perceived by the students, however 
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these items were eliminated by the AIT staff. A major shortcoming 
of the questionnaires was the lack of in-depth information about 
problems identified by the respondents. The case studies contri- 
bute substantially to correcting this situation by enabling AIT 
staff to collect in-depth information about PT curriculum prob- 
lems. The cost of conducting the case studies has limited the 
number of sites and the time spent on-site, but the information 
gained from a limited number of case studies greatly enriches the 
formative evaluation data base. 

Propriety , Respondents were assigned numbers , anu AIT staff 
members exercised considerable caution in maintaining the ano- 
nymity of sites and individual respondents. Cate was exercised so 
that excessive time would not be involved in completing the evalu- 
ation questionnaires. Open-ended questions allowed respondents 
the opportunity to identify strengths and weaknesses of the PT 
curriculum. 

Accuracy , Administrators, teachers, and students were aware 
of and understood the purpose of the formative evaluation activ- 
ities. As would be expected, some respondents were more enthus- 
iastic than others regarding the evaluation activities, but there 
did not seem to be any threat to the accuracy of the evaluative 
responses. The formative evaluation objectives and activities 
were presented with sufficient clarity to permit their replication 
by other evaluators, should such a need arise. 

Developer implementation . The AIT/ CORD staff conducted a 
workshop for teachers designated to teach Principles of Technol- 
ogy. The evaluation activities and procedures were described 
during the workshop. In addition, teachers were mailed packets of 
questionnaires throughout the year. Each of these packets in- 
cluded explanatory material concerning the use of the instruments. 
Teachers were well-informed of the purposes of the formative 
evaluation and of the procedures for administering the question- 
naires. 

Conclusions concerning the formative evaluation of the Prin- 
ciples of Technology curriculum materials are presented in the 
following chapter. Recommendations are also presented regarding 
the planning and implementation of formative evaluation activi- 
ties. 
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CHAPTER 5 



CONCLUSIONS AND RECOMMENDATIONS 



The conclusions and recommendations presented in this report 
are based on task staff involvement with the formative evaluation 
of the Principles of Technology curriculum materials. 



Assessment Conclusions 

The following conclusions are based on (1) task staff 
involvement throughout the year with the formative evaluation of 
the Principles of Technology curriculum materials, (2) information 
collected from the ongoing AIT formative evaluation of the PT 
curriculum materials, and (3) information collected from AIT staff 
and local pilot site personnel as they were implementing the 
revised formative evaluation procedures and instruments. 

o The AIT evaluation staff was receptive to suggestions for 
improving the formative evaluation effort, but it was 
extremely difficult or impracticable to make certain 
changes after the evaluation system had been operating for 
a full year. 

o Individuals at State and local sites understood the need 
for the formative evaluation and were receptive to changes 
in the evaluation that would result in better information 
for curriculum revision. However, there is a limit to the 
amount of time that evaluators can reasonably expect local 
and State personnel to devote to evaluation purposes, 
particularly when teachers are a major source of evaluative 
information and also have the major responsibility for 
implementing a new curriculum. 

o Among those individuals with the evaluation effort, there 
were no special concerns raised concerning legal, ethical, 
and common sense considerations. 

o The most useful information collected by using the evalua- 
tion instruments came from open-ended questions addressed 
to teachers and students. The open-ended questions allowed 
teachers and students to explain why something was or was 
not working. 

o The approach being used to evaluate the Principles of 
Technology curriculum materials provided a balance of 
positive and negative feedback to curriculum developers. 
The addition of case studies provided an opportunity for 
probing and in-depth description. 
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o In general , most individuals understood the major purpose 
of the formative evaluation and were willing to devote 
sufficient time so that the evaluation resulted in meaning- 
ful information for curriculum revision. 

o The evaluation approach seemed to provide a sense of bal- 
ance concerning the positive and negative aspects of the PT 
curriculum. 

o The Principles of Technology curriculum was operating in 
complex and diverse settings. The formative evaluation 
approach did not provide sufficient information about the 
complexity of individual sites. The case studies partially 
remedied this situation, but their implementation did not 
occur until the second year of the field test of PT curri- 
culum materials. 

o It was probably a good public relations move to collect 
formative evaluation information from all PT sites. This 
enabled each State to develop a feeling of some ownership 
of the evaluation information. However, from the stand- 
point of cost and quality of data collected, it would have 
been more desirable to have used a small sampie (e.g., one- 
third of sites) and do more m-depth probing at selected 
sites. 



Recommendations for Formative 
Evaluation of Curri culum Efforts 

The following recommendations are based on task staff 
involvement with the design for the formative evaluation of the 
Principles of Technology curriculum effort, previous evaluation 
efforts, and reviews of relevant literature. 

o The designing of a formative evaluation effort should be 
done concurrently with the development of the curriculum 
and the development of strategies for field testing the 
curriculum. 

o The purpose and nature of formative evaluation should be 
thoroughly explained for all concerned parties so that 
expectations for results do not overburden or undercut the 
evaluation effort. 

o The information needed for curriculum revision should be 
clearly identified at the time the curriculum development 
is being planned so that sufficient consideration can be 
given to planning and implementing the evaluation effort. 
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o A combination of quantitative and qualitative data will 
most likely provide curriculum developers with the informa- 
tion they need for revision purposes* 

o Evaluation procedures should be designed so that key indi- 
viduals (e.g., students, teachers, and selected others) 
have an opportunity to provide information about the value 
of the curriculum* 

o The curriculum developer/evaluator and the curriculum users 
(teachers, students) should make use of the telephone or 
other means that permit two-way communication. However, 
this should not be construed as an acceptable substitute 
for on-site, in-depth observations, interviews, and 
document/record reviews. 
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PRINCIPLES OF TECHNOLOGY 
Instructional Package for Unit 3: RATE 
Reviewer Questionnaire 



SECTION I - STUDENT HANDBOOK 
1. What did you like most about the student handbook? Why? 



2. what did you like least about the student handbook? Why? 



3. Is the student handbook written at the appropriate level for 
11th and 12th-grade vocational students? 

definitely probably not 

probably definitely not 



Comments: 



4. Are the examples and illustrations used in the student 

handbook relevant to 11th and 12-grade vocational students? 

definitely probably not 

probably definitely not 



5. Are there any errors or inaccuracies in the student 
handbook? 

yes no 



If yes, please specify, including page number: 
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6, Is the content of the math labs useful for this unit? 

definitely probably not 

probably definitely not 



7. Is the content of the hands-on-labs useful for this unit? 

definitely probably not 

prok ibly definitely not 



8. Do you have any other comments, concerns, or suggestions for 
the student handbook? 



SECTION II ■ TEACHER'S GUIDE 

1. Is the format of the teacher's guide (pages inserted nexu to 
the pertinent page of the student handbook) appropriate for 
the PRINCIPLES OF TECHNOLOGY course? 

yes, definitely no, probably not 

yes, probably no, definitely not 

If no, what forr.iat modifications would you suggest? 



2. Is the information presented in the teacher's guide 
accurate? 

yes no 



If no, list inaccuracies including page numbers 
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3. will the teacher's guide enable a teacher to successfully 
implement the unit? 

definitely probably not 

probably definitely not 

don't know 



If not, why not? 



4. Do you have any other comments, concerns, or suggestions for 
the teacher's guide? 



SECTION III - VTDEO PROGRAMS 

1. Which video program did you like the mzsL? Why? 

2. Which video program did you like the le_aa£? Why? 

3. Are the examples used in the video programs relevant to the 
llth-and 12th-grade vocational students? 

yes no 



If no, list your concerns. Do you have suogestions for 
alternative examples? if so, please list below: 

4. Are the examples used in the video programs useful for this 
unit? 

yes no 



If no, list your concerns and suggest alternative examples: 
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5. Is the content of the video programs accuurate? 
yes no 



If no, list your concerns: 



SECTION IV - OVERALL 

6. Do the student handbook and video programs augment one 
another? 

yes no 



Comments: 



2. Will the materials (student handbook and video programs) 

increase students 1 understanding of RATE ? 

deTinitely probably not 

probably definitely not 



3. Overall, how would you compare the RATE unit to units 1 and 
2? 

better worse about the same 



If worse, why? 
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In terms of tneir overall impact (instructional 
effectiveness, student interest, teacher manageability) rank 
of each of the components of the RATE unti using the 
following scale: 

A = Excellent 

B « Good 

C = So-so 

D = Poor 

E = Terrible 

Place the letter, corresponding to your ranking, next to each 
component 

Student Handbook hands on labs 

Video teacher's guide 

math labs 

Please explain any C, D, or E rankings: 



5, Do you have any other comments, concerns, or suggestions for 
the unit on RATE? 



Name. 

Consortium Agency 
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PRINCIPLES OP TECHNOLOGY 
UNIT ? : RATE 
STUDENT ATTITUDE QUESTIONNAIRE 

Sex: Female Male 

Grade: 9 10 11 12 

1. Overall, did you like the unit on RATE? 

yes, a lot nc not very much 

yes, a little no, not at all 

2. What component did you like most in the RATE unit? 

the written material the hands on labs 

the video programs no preference 

the math labs 



4. Overallr was the material that was covered in the RATE unit 
difficult for you to understand? 

yes, most of the material was difficult for me to 

understand 

yes, some of the material was difficult for me to 

understand 

no, most of the material was not difficult for me to 

understand 

5. Which component of the RATE unit was the meat difficult for 
you to understand? 

the written material the hands on labs 

the video programs no component was 

the math labs particularly difficult 

7. Do you think the material in the RATE unit is important for 
you to understand? 

yes, very important no, not very important 

yes, sort- of important no, not at all important 

8. Do you have any comments about the RATE unit? 



THANK YOU! 
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PRINCIPLES OF TECHNOLOGY 
STUDENT BACKGROUND QUESTIONNAIRE 



Grade: 9 10 11 12 

Sex: Female Male 

1. What is your area of concentration in vocational education? 

electronics metal working 

hydraulics/pneumatics general industrial arts 

auto mechanics other (please specify): 

woodworking r ■* 

agriculture/horticulture 

am not in vocational 

education 

do not have an area of 

concentration 

2. Which of the following math cours - have you taken or are 
currently taking (check all that apply)? 

general math trigonometry 

algebra calculus 

geometry other (please specify): 



3. which of the following science courses have you taken or are 
currently taking (check all that apply)? 

physical science chemistry 

general science physics 

biology other (please specify) : 



4. what do you plan to do after high school? 
get a job 

go to a vocational school 

go to a junior college 

go to college 

I don't know 



THANK YOU! 
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PRINCIPLES OF TECHNOLOGY 
UNIT III: RATE 
STUDENT TEST 

1. In the unifying definition for rate, rate is the relationship 
between 

a. forcelike quantity and distance 

b. work and elapsed time 

c. displacementlike quantity and elapsed time 
d„ voltage and charge 

In the left column are rates that may or may not exist in one of 
the energy systems described in the right column. On your answer 
sheet fill in the letter of the energy system that corresponds to 
the numbered rate. 



2. 


Mass flow (m/t) A, Rate in 


mechanical pnerav f ?vshpm^ 


3 . 


Current (a/t) B. Rate in 


"F 1 11 1 H pnp r rt\7 CVC f PTTlC 


4. 


Acceleration (v/t) C. Rate in 


electrical pn&ra\z <?v<5fpni^ 


5 . 


Heat How per D. Rate in 


thermal energy systems 




plan^pd f i mp [\\/\-\ 

cia^ocu l. x lite V **/ L. ) 




E. Is not 


a rate in an energy system 


6 . 


Postal rate (cents/ounce) 




7 . 


Angular speed (0/t) 




8. 


When using the Internal System 


of units (SI), mechanical 




is measured in 






a. newton meters (N.m) 






b. newtons per second (N/s) 






c. feet per second (ft/s) 






d. meters per second (m/s) 




9. 


Linear speed is 






a. distance an object travels 


along a line in a unit of 




time 






b. always expressed in SI units 




c. must include magnitude and 


direction 




d. measured in radians per second 
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10. Which of the following is a linear rate ? 

a. helicopter blades in flight 

b. hands moving on a clock 

c. a jet airplane accelerating down a runway 

d. all of the above 

11. Which of the following involves ^n an^lar rqt e? 

a. helicopter blades in flight 

b. ferris wheel 

c. hands moving on a clock 

d. all of the above 

12. An increase in linear speed during an elapsed time is 
described as 

a. an average speed 

b. an acceleration 

c. a deceleration 

d. a new linear speed 

13. Angular speed is 

a. always expressed in SI units 

b. measured in radians per unit of time 

c. distance an object travels along a line in a unit of 
time 

d. a vector quantity 

14. when changed to radians per second, 4 RPM's is equal to 
(Note: Remember that 1 minute = 60 seconds and 2 radians = 
360. 

a. 0.210 rad/sec 

b. 0.418 rad/sec 

c. 0.636 rad/sec 

d. 1.272 rad/sec 
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15. A unit of rate associated with a fluid system is 

a. gallons per minute 

b. kilograms per second 

c. cubic feet per minute 

d. all of the above 

16. Volume flow rate is 

a. volume moved per unit of time 

b. determined by the length of the pipe 

c. always expressed in SI units 

d. a linear rate 

17. A car air conditioner pumps 5 kg of freon gas through the 
system in 3 minutes. The mass flow rate of the fre^n gas is 

a. .60 kg/min 

b. 1.65 kg/min 

c. 15 kg/min 

d. not enough information given to calculate 

18. One coulomb per second is a unit of 

a. electrical charge 

b. electrical voltage 

c. electrical current 

d. electrical resistance 

An AC current changes direction at the rate of 120 times per 
second. Use this information to answer questions 19 and 20. 
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19. The frequency of the current is 

a. 1/120 cycles per second 

b. 1/60 cycles per second 

c. 120 cycles per second 

d. 60 cycles per second 

20. The period of the current is 

a. 1/120 seconds 

b. 1/60 seconds 

c. 120 seconds 

d. 60 seconds 

21. When an ammeter is placed in a circuit to measure current 

through a resistor, it is placed in with the 

resistor 

a. series 

b. parallel 

c. at right angles 

d. none of the above 

22. The term 60 Hz is an electrical 

a. period 

b. frequency 

c. event 

d. amplitude 

23. Rate in an electrical system is 

a. how fast charge flows through a conductor in a unit of 
time 

b. electrical current in coulombs 

c. only measured in DC current 

d. all of the above 
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24. Heat flow rate is 

a. calories per second 

b. BTUs per minute 

c. heat energy per unit of time 

d. all of the above 

25. In the English system, heat flow rate is stated in 

a. calories/second 

b. joules/ second 

c. BTUs/hour 

d. all of the above 

26. The amount of heat energy required to raise the temperature 
of a given body one degree is 

a. latent heat 

b. heat capacity 

c. specific heat 

d. sensible heat 

27. How many units of heat energy are required to raise a unit 
mass a unit temperature difference defines 

a. latent heat 

b. heat capacity 

c. specific heat 

d. sensible heat 

28. The heat capacity of 10 pounds of water is 

a. 10 cal/C 

b. 1 BTU/F 

c. 10 BTU/hr 

d. 10 BTU/F 

6C 
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29. The calories per second of heat energy flowing through a 
window pane does not depend on 

a. thickness of glass 

b. weight of the glass 

c« surface area of the glass 
d. type of glass 

30. Water at 90 C to steam at 100 C is 

a. latent heat 

b. sensible heat 

c. both latent and sensible heat 

d. specific heat 
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PRINCIPLES OP TECHNOLOGY 
TEACHER PERCEPTIONS OP STUDENT CHARACTERISTICS 

1. How many students are in your PRINCIPLES OF TECHNOLOGY 
class? 

2. Have you had any of these students in classes other than 
PRINCIPLES OF TECHNOLOGY? yes no 

If yes, which classes? 



3. Are the students in your PRINCIPLES OF TECHNOLOGY class all 
from one school district? yes no 

If no, how many school districts do they come from? 
2 '\ 4 5 ipore than 5 

4. Please briefly describe how students were selected for your 
PRINCIPLES OF TECHNOLOGY class: 



5. What is the ability level of your students? Please indicate 
the approximate number of students in each category: 

above average average below average 



Comments : 



What is the socioeconomic level of your students? Please 
indicate the a pproximate number of students in each 
category : 

upper class lower middle class 

upper middle class lower 

middle class 



7. What is the racial/ethnic composition of the class? Please 
specify the number of each: 

White Oriental 

Black American Indian 

Hispanic Other (please specify): 



8. Please detail any other information about your class that you 
feel is pertinent. 
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PRINCIPLES OF TECHNOLOGY 
UNIT III: RATE 
TEACHER QUESTIONNAIRE 



What did you like mos_t about the RATE unit? 



What did you like le^ai about the RATE unit? 



Overall, how would you compare the RATE unit to units 1 and 



better about the same 



worse 



If worse, why? 



In terms of their overall impact (instructional 
effectiveness, student interest, manageability) rank each of 
the components of the RATE unit using the following scale: 

A = Excellent 
B = Good 
C = So-so 
D = Poor 
E = Terrible 

Place the letter, corresponding to your ranking, next to each 
component. 

student handbook hands on labs 

videos teacher's guide 

math labs 



Please explain any C, D, or E rankings an.l/or list any other 
comments you have about the components: 



Which of your students seem to be the most successful in the 
PRINCIPLES OF TECHNOLOGY course? 

above average below average 
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6. Based on your experiences , do you think the 6-day plan, per 
subunit, of 50-minute class sessions is realistic for the 
subunits? 

^ yes, definitely no, probably not 

yes, probably no, definitely not 

If no, please explain: 



7. On average, how much time did you spend preparing to teach 
ea^h class in the unit on RATE? 



0-30 minutes 
31-60 minutes 
61-90 minutes 



91-120 minutes 
121-180 minutes 
181 or nore minutes 



Comments: 



8. Overall, did you feel comfortable teaching the materials in 
the unit on RATE? 



yes, very comfortable 
yes, sort of 
comfortable 



no, not very coiu.or table 
no, i*ot at all 
comf ortc ble 



Ir no, please specify: 



9. Do you think moist of your students did the assigned readings 
at home? 



definitely 

definitely not 



probably 



probably not 



Comments: 



10. What, if anything, caused you i,he most problems in teaching 
the unit on RATE? 



11. Do you feel the Teacher's Guide material provided you with 
enough information to help you successfully implement the 
unit? 
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12. Did you teach Unit- III: RATE on consecutive days for 26 
days? 

yes no 



If no, what pattern did you use (for example, 3 days a 
week)? 



13. How much time per session did you teach? 

50 minutes or less 61-90 minutes 

51-60 minutes 91+ minutes 



14. Did you combine any classes into one session (for example, 
teach classes CI and C2 in one session)? 



yes no 



If yes, which classes did you combine? 



15. How many physics courses did you take in college 
(undergraduate and graduate)? 



none 5-7 

1 0 or more 

2-4 



16. 



How many math courses did you take in college (underqraduate 
and graduate)? 



none 5-7 

1 8 or more 

2-4 



17. Do you have any other comments, concerns, or suggestions for 
the unit on RATE? 

The following chart lists each activity for t<ie unit on RATE 
down the left column. Since there are no materials specifically 
for the sub-unit review classes, tnese classes have not been 
listed in the chart. For each activity, you should respond to the 
following questions by circling "yes" or "no" : 



1. 



Was the material (readings, labs, or videos) appropriate 
for your students? Was the material at the right grade 
level? Was the amount of material appropriate for your 
students? For any no r sponses. please use the attached 
pages to describe your concerns. 
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2. were you able to cover the material to your satisfaction 
in the 50-minute time period? (Since this question 
doesn't apply to the video, no response options have 
been provided for the column. Please do respond, 
however, to the other questions about the video.) For 
any no r esponses , please use the attached pages to 
describe why you could not complete the material and/>r 
what you chose to delete. 

3. Were there any errors or inaccuracies in the material? 
For any yes responses , use the attached pages to specify 
the errors and recommended corrections. 

4. Were there any problems managing the activity? For the 
labs, were all your students able to rotate through the 
labs? Did you experience any problems coordinating the 
labs? Did you experience any problems setting up or 
tearing down the labs? Did you experience any problems 
coordinating the activity? For any yes responses , use 
the attached pages to specify the problems you had and, 
if possible, suggest changes that you feel would enable 
you to more easily manage the material. 

5. Do you have any suggested modifications for the 
material? For any yes responses , use the additional 
pages to specify your suggestions. Include in this 
section any "teaching tips" — special procedures you used 
or means you discovered to more easily convey the 
information to students. Include in this section any 
comments you may have for the Teacher's Guide. 

We recommend that you take a few minutes each day to complete 
the chart and, most importantly, to write down your comments. If 
you need more space for comments, use the back of the comments 
Waqes and/or attach additional sheets. 




UNIT III: RATE 





Appropriate 
for students? 


Completed 1n 
50 minutee? 


Any errors or 
inaccuracies? 


Any management 
probl erne? 


Any suggeeted 
modi Ticationsr 


Overview 
Video 


yes no 


— 


yes no 


yee no 


yee no 


CO 


yes no 


yes no 


yes no 


yes no 


yee no 


Hecheniral Systems 












Video 


yee no 


— 


yee no 


yes no 


yee no 


d 


yes no 


yes no 


yee no 


yes no 


yes no 


C2 


yee no 


yee no 


yes no 


yee no 


yes no 














Hath Leb 


yee no 


yes no 


yee no 


yes no 


yes no 


3 Ml 


yet no 


yee no 


yee no 


, as no 


yes no 


3M2 


yes no 


yee no 


yee no 


yee no 


yes no 


Fluid Systems 












Video 


yea no 


— 


yes no 


yee no 


yes no 


CI 


yes no 


yee no 


yee no 


yee no 


yes no 


C2 


yes no 


yee no 


yee no 


yes no 


yee no 


Hath Leb 


yes no 


yes no 


yes no 


yee no 


yee no 


3F1 


yea no 


yes no 


yee no 


yes no 


yee no 


3F2 


y<*8 no 


yes no 


yee no 


yes no 


yes no 


Electrical Systeme 












Video 


yes no 




yes no 


yes no 


yes no 


CI 


yee no 


yes no 


yee no 


yee no 


yes no 


C2 


yee no 


yee no 


yes no 


yee no 


yee no 
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UNIT III (continued] 



Hath Leb 


yes no 


yes no 


yes no 


yes no 


yes no 


3 El 


yes no 


yes no 


yes no 


yes no 


yes no 


3E2 


yes no 


yes no 


yes no 


yes no 


yes no 


Th?rmel Systems 












Video 


yes no 


— — 


yos no 


yes no 


yes no 


CI 


yes no 


yes no 


yes no 


yes no 


yes no 


C8 


yes no 


yes no 


yes no 


yes no 


yes no 


Moth Leb 


yes no 


yes no 


yes no 


yes no 


yes no 


3T1 


yns no 


yes no 


yr c no 


yes no 


yes no 


3T2 


yes no 


yee no 


yes no 


yes no 


yes no 


Summery 












Video 


yes no 


yee no 


yes no 


yee no 


yes no 
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Comments for Over, .lew Class 
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Comments for Videos 

Overview Video: 
Mechanical Systems Video: 
Fluid Systems Video: 
Electrical Systems Video: 
Thermal Systems Video: 
Summary Video: 
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Comments for CI Classes 
Mechanical Systems CI: 

Fluid Systems CI: 

Electrical Systems CI: 

Thermal Systems CI: 
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Comments for C2 Classes 
Mechanical Systems C2 : 

Fluid Systems C2 : 

Electrical Systems C2 : 

Thermal Systems C2: 
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Comments for Math Lab Classes 
Mechanical Systems Math Lab: 

Fluid Systems Math Lab: 

Electrical Systems Math Lab: 

Thermal Systems Math Lab: 
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Comments for Lab 1 Classes 
Mechanical Systems Lab 1: 

Fluid Systems Lab 1: 

Electrical Systems Lab 1: 

Thermal Systems Lab 1: 
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Comments »r Lab 2 Classes 
Mechanical Systems Lab 2: 

Fluid Systems Lab 2: 

Electrical Systems Lab 2: 

Thermal Systems Lab 2: 
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Video Production 



Teacher Teacher 

W„ r k s ho p 1 Wo r k e ho p 2 

Pilot Test 1-7 Pilot Test 8-14 

I I L.J ... 

c::j c::j 

Consortium Consortium 

Heetlng 1 Meeting 2 



Information 
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Release 
1 -7 
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APPENDIX D 



THE NATIONAL CEN r "R SUGGESTED APPROACH 
FOR THE SECOND- YEAK FORMATP E EVALUATION 
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Unit Number 



PRINCIPLES OF TECHNOLOGY UNIT EVALUATION 
STUDENT REACTIONNAIRE 



PJifiEOSE: 

Your answers to the questions on this form v .11 help improve this 
Principles of Technology unit for other students. Your responses 
to the following items are appreciated. 
DIRECTIONS : 

Below are a list of statements. Flease read each statement and 
answer whether you strongly agree, agree, disagree, or strongly 
di-3gree by checking the appropriate box. Place a check mark ( ) 
in only one box for each statement. 



Strongly 
Agree 



Strongly 

Agree Disagree Disagree 



UNIT OBJECTIVE S 



For this unit: 



(1) The objectives were 
clear on what I was 
to learn 



( ) 



( ) 



'2) I knew what I needed to 
do to achieve the 
objectives 



( ) 



(3) I knew when I had 
achieved a unit 
objective. . . . 



( ) 



( ) 



( ) 
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Strongly Strongly 
Agree Agree Disagree Disagree 



ERIC 



(4) What I learned and what 

I was supposed to learn 
(as stated by the ob- 
jectives) were dif- 
ferent ( ) ( ) ( ) ( ) 

UNIT CONTENT 

The content of this unit: 

(5) Was presented in a 

logical order. ... ( ) ( ) { ) ( ) 

(6) Had too much informa- 
tion ( ) ( ) ( ) ( ) 

(7) Used language difficult 

for me to understand ( ) ( ) ( ) ( ) 

(8) Gave examples helpful 
in understanding major 

concepts ( ) ( ) ( ) ( ) 

(9) Was difficult for me 

to understand. ... ( ) ( ) ( ) ( ) 

(10) Will probably be useful 

in future jobs ... ( ) ( ) ( ) ( j 

(11) Did not have enough 

examples ( ) ( ) ( ) ( ) 

UNIT P OSTTEST 

Tha posttest questions: 

(12) Were covered in the 

unit instruction . . ( ) ( ) ( ) ( ) 

(13; were clearly stated ( ) ( ) ( ) ( ) 

(14) Sampled important 

points in the unit { ) ( ) ( ) ( ) 

(15) Were generally fair ( ) ( ) ( ) ( ) 
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Strongly Strongly 
Agree Agree Disagree Disagree 



UNIT VIDEO 

The video programs for this 
unit: 

(16) Supported the text 

material ( ) ( ) ( ) { ) 

(17) Used easy to under- 
stand graphics . . ( ) ( ) ( ) ( ) 

(18) used language dif- 
ficult for me to 

understand .... ( ) ( ) ( ) ( j 

(19) Used theme music that 

1 liked ( ) ( ) ( ) ( ) 

m 

The unit text materials: 

(20) Helped me to achieve 

the unit objective** ( ) ( ) ( ) ( ) 

(21) Will be a useful refe- 
rence aft-er taking 

the course .... ( ) ( ) ( ) ( j 

(22) Used language difficult 

(23) Had enough examples to 
help me understand the 

important concepts ( } ( ) ( } ( ) 

LABORATORY LQIIXIXZZS 
The unit lab activities: 

(24) Helped ma to achieve 

the unit objectives ( ) ( ) { ) ( 

(25) Were difficult . . ( ) ( ; ( * ( ) 

(2*) Should not include 

math labs ( ) ( ; ( ) ( j 
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Strongly Strongly 
Agree Agree Disagree Disagree 



(27) Take too long to 
set up the equip- 
ment ( ) ( ) ( ) ( ) 

(28) Time periods wero 
not long enough to 

complete the work ( ) ( ) ( ) ( ) 

(29) In general, were 

well like ( ) ( ) ( ) ( ) 

UNIT DIFFICULTY 
For this unit: 

(30) I did not have 
enough time to do 

my work ( ) ( ) ( ) ( ) 

(31) Some of the thinys 
the teach wanted 
me to learn were 

just too hard . . ( ) ( ) ( ) ( ) 

(32) I had trouble 
reading the unit 

text materials. . ( ) ( ) ( ) ( ) 

(33) The teacher gave 
me too much work 

to do ( ) ; ) { ) ( ) 

TUDENT SATISFACTION 
For this unit: 

(34) I usually had a 
sense of satis- 
faction after 
leaving class 

each day ( ) ( ) ( ) { ) 

(35) I did not like 

coming to class ( ) ( ) ( ) ( ) 
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Strongly Strongiy 
Agree Agree Disagree Disagree 



(36) My classmates felt 
good about what 

happened in class ( ) ( ) ( ) ( ) 

(37) I felt good about 
what happened in 

cl * ss ( ) ( ) ( ) ( ) 

(38) I would not recom- 
mend it to my 

friends ( ) ( ) ( ) ( j 
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Unit Number: 

PRINCIPLES OF T ECHNOLOGY UNIT EVALUATION ; 
TEACHER REACTIONNAIRE 

PURPOSE ; 

The primary purpose of this reactionnaire is to provide teacher 
information for facilitating improvement of the Principles of 
Technology curriculum. Specifically, this form is designed to 
assess the appropriateness of Principles of Technology unit 
objectives and the appropriateness of instructional activities, 
resources, curriculum content, posttests, and instructional 
planning needed to accomplish these objectives. 

DIRECTIONS : 

Below a-e a list of statements. Please read each statement and 
answer whether you strongly agree, agree, disagree, or strongly 
disagree by checking the appropriate box. Place a check mark ( ) 
in only one box for each statement. 



UNIT OBJECTIVES 

The unit objectives : 



Strongly Strongly 
Agree Agree Disagree Disagree 



(1) assist me in assessing 
student progress . . . 



( ) 



( ) 



( ) 



( ) 



(2) The unit objectives 
are difficult for 
me to use 



( ) 



( ) 



( ) 



(3) don't reflect what 
the unit is designed 
to teach 



( ) 



( ) 



( ) 



( ) 



(4) assist students in 
knowing what is 

expected of them ... ( ) 

(5) are worded too 
sim 1 j stic to be 

of value ( ) 



( ) 



( ) 



( ) 



( ) 



( ) 



( ) 
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Strongly Strongly 
Agree Agree Disagree Disagree 



(6) help me Jetermine 

what to teach ( ) ( ) ( j ^ j 

(7) are presented in a 

logical order ( ) ( ) ( ) ( } 

UNIT INSTRUCTIONAL ACTTVTTTEft 

The unit instructional activities: 

(8) are related to unit 

objectives ( ) ( } ( } ( } 

(9) require mote than one 
instructor to teach 

the unit content . . . ( ) ( ) ( ) ( j 

(10) are presented in 

the correct sequence ( ) ( ) ( ) ( j 

(11) require a reading 
level that is too 
difficult for most 

students ( ) ( j ( j ( ) 

(12) do not have enough 

math labs { ) ( j ( j ^ j 

(13) take too much time 
for most students to 

complete ( ) ( } ( } ( } 

UNIT CONTENT 

The unit content ; 

(14) was presented in a 

logical order ( ) ( j ( j ^ j 

(15) was too detailed . . . ( ) ( ) ( ) ( j 

(16) was too easy ( ) ( j ^ j ^ j 

(17) was just right ....() ( ) ( ) ( } 
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Strongly Strongly 
Agree Agree Disagree Disagree 



(18) used language that 
was difficult for 
most students to 

understand ( ) ( ) ( ) ( ^ 

(19) contained examples 
that were helpful to 
students in their 
understanding of 

unit concepts ( ) ( ) ( ) ( ) 

(20) contained enough 

examples ( ) ( ) ( ) ( ) 

(21) provided enough sum- 
aries of important 

points ( ) ( ) ( ) ( ) 

(22) provided information 
that will be useful 
for students in their 

future employment. . . ( ) ( ) ( ) ( ) 

UNIT POSTTEST 

The unit posttest questions: 

(23) asked different things 
than what had been 

taught ( ) ( ) ( ) ( ) 

(24) were clearly worked. . ( ) ( ) ( ) ( ) 

(25) covered all the impor- 
tant point in the unit ( ) ( ) ( ) ( ) 

(26) werf. generally fair. ( ) ( ) ( ) ( ) 
PERCE -TONS OF INSTRU CTIONAL PLANK I KG 

(27) How many paid hours of plannino (e.g., planning periods, 
after school) did you receive for planning and preparing 
materials for this unit? 

Number of hours 
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Is this amount of time adequate? 
( ) Yes 

( ) No, I need additional hours. 

(28) Approximately how much time did you expect students to spend 
on homework each day for this unit? 

( ) None 

( ) About half an hour 

( ) About one hour 

( ) About two hours 

( ) More than two hours 

(29) What percentage of students typically completed your homework 
assignments for this unit? 

( ) Fewer than 25% ( ) 50% fo 74% 

( ) 26% to 49% ( ) 75% and above 

(30) Do you feel the Teacher's Guide material p.-^vided you with 
enough information to help you teach the unit? 

( ) Definitely ( ) Probably ( ) Probably not ( ) Definitely 

(31) Did you teach this UNIT on consecutive days for 26 days? 

( ) Yes ( ) No 

If no, what pat-ern did you see (for example, 3 days a week)? 

(32) How much time per class section did you devote to this UNIT? 

( ) 50 minutes or less ( ) 61-90 minutes 

( ) 51-60 minutes ( ) 91+ minutes 

(33) Did you combine any of the unit's classes into one session 
(for example, teach classes C.I and C2 in one session)? 

( ) Yes ( ) No 

If yes, which classes did you combine? 

(34) Based on >our experiences, do you think the 6-day plan, per 
sub-unit, of 50-minute class sesrions is realistic for this 
Unit? 
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( ) Yes, definitely ( ) No, probably not 

( ) Yes, probably ( ) No, definitely net 

(35) On average, how much time did you spend preparing to teach 
each class in this Unic? 

( ) 1/4 hour ( ) 1 hour 

( ) 1/2 hour ( ) 1 1/2 hours 

( ) 3/4 hour ( ) 2 hours or more 

Comments: 

(36) Overall, did you feel comfortable teaching the materials in 
this UNIT? 

( ) Yes, very comfortable ( ) No, not very comfortable 

( ) Yes, sort of comfortable ( ) No, not at all comfortable 
If no, please explain: 



(37) what* if anything, caused you the most problems in teaching 
this UNIT? 



(38) What did you like the most in teaching this UNIT? 
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UNIT 

PRINCIPLES OF TECHNOLOGY 
UNIT DAILY LOG 

The purpose of the Unit Daily Log is to provide a continuous 
source of teacher information for improving the Principles 
of Technology curriculum. 

DIRECTIONS ; 

The left column on the following charts list each activity for 
this particular unit. Since there are no materials specifically 
for the sub-unit review classes , these classes have not been 
listed in the chart. 

For columns 1-5 on the attached charts: 

(1) Circle tt Y B (yes) or w v" (No): Were the reading, labs, 
or videos appropriate (e.g., grade level, sufficient 
quantity of material) for your students? if not, 
specify modifications in column 5. 

(2) Circle B Y" (yes) or "N" (No): Were you able to cover 
the content in the 50-minute time period? 

(3) Briefly describe any errors or inaccuracies in the 
material. 

(4) Briefly describe any problems you had in managing the 
material. Amoiig others, this may include problems in 
coordinating lab rotations, lab set-up, and maintaining 
student interest. 



(5) 



Briefly list suggestions for modifying the material. 
Describe "teaching tips" for future teachers. 



We recommend that you take a few minutes each day to complete the 
charts. If you need more space for comments, use the back of the 
charts and/or attached additional pages. 
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CASE STUDIES 



PRINCIPLES OF TECHNOLOGY 



Setting Boundaries 



The primary purpose of the case studies is to provide 
information for facilitating improvement in the Principles of 
Technology curriculum. In view of the primary purpose of the case 
studies the investigation will be limited to secondary school 
sites participating in the Principles of Technology pr ject. The 
major focus at the secondary school site will be the Principles of 
Technology class. 



Selecting Sites 

Purposeful sampling is to be done in selecting the sites. 
Six sites will be studied. In order to achieve a wide variety of 
conditionsr sites will be selected according to the following 
criteria: 



o exceptionally good programs 
o typical programs 
o geographical variation 
- areas of the country 
rural , urban, suburban 
o programs serving special reeds groups 
o convenience 

Caution will be exercised by not selecting politically sensi- 
tive sites. 



Establishing Initial Contacts 

Once sites have been selected, appropriate approvals will be 
secured. The protocol network will be followed in securing 
approval to conduct the case studies. While the protocol may vary 
by state r in general, the procedure will involve the following: 

o state agency approval 
o local agency approval 
o local teacher approval 

Confidentiality will be assured with regard to the 
identification of state and local sites and individuals. 

D -N/eloping Data Collection Procedures 

Data will be collected by: 

o Interviews - The interviews are designed to provide *"n- 
depth qualitative descriptions of Principles of Te< biology 
teacher, student, and administrator perceptions on the 
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following topics: objectives, content, learner activi- 
ties, teacher knowledge, and instructional techniques, 
posttests, instructional aids, equipment and materials, 
and an overall assessment of the Principles of Technology 
curriculum, in addition, where appropiiate, interviews 
will be conducted with representatives of business/ 
industry and members of citizen advisory committees. 

o Qbservat ilojis. - The observations are designed to provide 
in-depth qualitative descriptions of Principles of Tech- 
nology classroom instruction. 

o Document /Re cord aejtifi&fi - The reviews of printed or writ- 
ten accounts should provide a picture on a stronger con- 
textual basis for what has happened in the Principles of 
Technology classes. 

0 Analysis of Rvtanh na , f a - The analysis of information from 
the student reactionnaire, teacher reactionnaire, the unit 
daily log, pretests, and posttests provides additional 
data input for the case studies. 

Potential Topics and fiuhtopics for Guiding 
Interviews. Observer i np.fi, Pocumgnr/Rernrfl 
Reviews and Data Recording Categoric 

These top-cs and subtopics can be used for focusing inter- 
views, observations, document/record reviews, ard the analysis of 
extant data. They can also serve as a framework for coding the 
data collected. 

I. Objectives 

A. Clarity of meaning 

B. Clarified whaL was to be learned 

C. Appropriate coverage of obje^ e in affective, psycho- 
motor, cor ,itive and percep' ■„ omains 

D. Level of difficulty of obj within each domain 

II. Content 

A. Logical order 

3. Correct level of dif' iculty (depth) 

C. Adequacy of amount or information 

D. Appropriate language level 

E. Meaningful examples 

F. Sufficient examples 

G. Clarity of directions 

H. Relevant 



ERIC 



101 



108 



III. 



Learner Activities 



ERIC 



A. Relevancy to objectives and content 

E. Appropriately sequenced 

C. Appropriate number 

D. Efficient use of time 

E. Problems created 



IV. Xzazh&L 



A. Adequacy of knowledge 

B. Effectiveness of instructional techniques 

C. Quick response to questions 

D. Availability to help 

E. Quality of student-teacher interaction 

V. Posttest 

A. Covered important points 

B. Clearly worded 

C. Fair 

VI. Instruct ional Aids 

A. Appropriate for objectives and content 

B. Reinforced concepts 

C. Interesting 

VII. Equipmen t and Materials 

A. Appropriate for objectives and content 

B. Difficulty of use 

C. Safety problems 

D. Difficult to locate 

E. Reasonable cost 

F. Complexity of construction and/or fabrication 

G. Instruction time required 

VIII. General 

A. What did you like most about course? 

B. what did you dislike most about course? 

C. Were parts of course boring? 

D. Were parts of course confusing? 

E. Amount or homework 

F. Any additional comments about course? 
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DESCRIPTION OF TOPICS AND SUBTOPICS 
(INTERVIEWS) 



I. Objectives 

Since the steps of the Principles of Technology instructional 
development process are driven by unit objectives, the first task 
is to determine the adequacy of these objectives. Interviews with 
teachers can provide reliable information about the appropriate- 
ness of the unit objectives in addition to providing evidence as 
to their worth. 

Specifically, teachers can provide useful information about the 
objectives' clarity of meaning. For instance, did the teachers 
and students understand the objectives? Do the objectives 
accurately reflect what the student is supposed to do or know' 
Can the objectives be operationalized? Moreover, teachers can 
determine if the unit objectives appropriately covered the 
affective, psychomotor, and cognitive domains. Teachers can also 
provide useful insights about the level of difficulty of the 
objectives within each of these domains. In sum, interviews with 
Principles of Tecnnology teachers should, in part, focus on 
obtaining information that can be used to judge the adequacy of 
unit objectives. H 1 

II. Content 

During the teacher interviews, information about the content of 
each Principles of Technology unit should be obtained. Teachers 
can provide useful content analysis input about the appropriate- 
ness of each unit. For example, it would be useful to know if the 
teachers believe that the unit content is presented in a logical 
order. Moreover, do the teachers believe that the unit is at the 

SESSS.J**? 1 ° f ^ if £ iCU i VY? 18 the lan 9 u *ge level of the units 
adequate? Insights to the adequacy of unit examples, graphics, 
and clarity of directions would also assist in the verification of 
unit content. 



III. Learner Activities 

During the teacher interview, information about the relevancy of 
the learning activities to unit objectives and content should be 
obtained. Are the math and hands-on laboratory activities appro- 
priately sequenced? Are the number of learning activities appro- 
priate? The teachers should be permitted to express their opin- 
ions about the effectiveness of the learning activities. Were the 
activities an efficient use of time? what classroom management 
problems did the teachers encounter with the learning activities? 
Teacher suggestions for improving the learning activities for each 
unit should also be obtained. 

103 

0 111 



ERIC 



IV. Teacher 



The teachers should be asked to comment on their knowledge of the 
unit subject matter. Is their knowledge adequate to teach the 
material? Teachers should be asked to comment on the effective- 
ness of the instructional techniques that are suggested in the 
teacher' s guide. Moreover, given the math and hands-on laboratory 
activities, can the teachers respond quickly to student questions? 
Likewise, during the laboratory activities, are teachers able to 
give sufficient individual student help? 



V. Posttest 

Teachers should be asked if the posttest covered the important 
points of the unit. Did the posttest ask different things from 
what had been taught? Were the posttest items clearly worded and 
were the items fair? Suggestions for improving the posttest 
should be obtained. 



VI. Instructional Aids 

Information about the appropriateness of the instructional aids 
should u e obtained during the teacher interview. Were the aids 
appropriate for the objectives and content? Did the video, stu- 
dent workbook, math exercises, and lab demonstrations reinforce 
unit concepts? Did the instructional aids maintain student 
interest? Suggestions for improving instructional aids should be 
obtained. 



VII. Equipment and Materials 

Information about the appropriateness of laboratory equipment and 
instructional materials should be obtained during the teacher in- 
terview. Were the equipment and materials appropriate for accom- 
plishing unit objectives? What procedural or management problems 
did the teachers encounter with the equipment and materials? Ask 
the teachers to nott any safety problems they discovered with the 
equipment. Were needed equipment and laboratory materials avail- 
able when it came time to use them? Information on the complexity 
of constructing laboratory set-ups and/or fabricating laboratory 
materials would be helpful in determining the difficulty of equip- 
ment and material use. Is the cost reasonable for unit equipment 
and materials? Was the time allotted for instruction sufficient 
to accomplish laboratory learning activities? 
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viii. General 

During the interview, teacher perceptions of what they liked and 
disliked most about the unit/course should be obtained. Teachers 
should note what parts oi unit/course were boring and confus- 
ing. Was the amount of homework required sufficient? Did stu- 
dents do their homework? Teachers should be permitted to make any 
additional comments about the unit or course as they see fit. 



DESCRIPTION OF TOPICS AND SUBTOPICS 
(OBSERVATIONS) 



I. Objectives 



During the classroom observation, observer impressions of unit ob- 
jectives should be noted. Did the students understand the impor- 
tance of the objectives? Were the objectives clear as to what the 
students were supposed to do or learn? The observer should note 
if there was an appropriate coverage of objectives in the affect- 
ive, psychomotor, and cognitive domains. Furthermore, within each 
of these domains, were the objectives appropriate as to their 
level of difficulty 0 Observer impressions of the usefulness of 
unit objectives in enhancing classroom learning should be noted. 

II. Content 

Observer impressions of the unit content should be noted. Speci- 
fically, did the unit appear to be at the correct level of diffi- 
culty? Did the language level appear to be appropriate for the 
majority of the students? Was the amount of information contained 
in the unit adequate? The observer should try to get a feel for 
the appropriateness of unit examples. Did the examples appear to 
be sufficient and meaningful? Moreover, did the students appear 
to understand directions that were stated by the teacher? In- 
sights on the clarity of student workbook directions would be 
helpful as well. 

III. Learner Activities 

Observer impressions on the relevancy of learning activities to 
objectives and unit content should be noted. Were the learning 
activities appropriately sequenced? Were there an appropriate 
number of activities planned? The observer should also note 
whether the learning activities were an efficient use of time. 
What problems did the instructor and students encounter durinq the 
learning activities that were observed? 
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iv. Teacher 



Did the instructor appear to have an adequate amount of knowledge 
of the subject area? Observer impressions of instructor's in- 
structional techniques should be noted. Was the instructor able 
to respond quickly to student questions? Was the instructor 
available to help all students who needed help? Overall impres- 
sions of the instructor 1 j command of the class may also be noted. 



V. Posttest 

Specif ically, the observer should attempt to determine if the unit 
posttest covered important points taught in class. Observer im- 
pressions of whether the posttest is fair and clearly worded 
should also be noted. 



VI. Instructional Aids 

Impressions of whether the instructional aids (video, graphics, 
math problems) are appropriate for unit objectives and content 
should be noted. Are the students interested in the instructional 
aids? Do the instructional aids appear to reinforce unit con- 
cepts? The observer should note any problems the students or 
teachers encountered when using the instructional aids. 



VII . Eq ui pment and Materials 

Observer impressions of the laboratory equipment and materials 
should be noted. Specifically, the observer should note if the 
equipment and materials are appropriate for unit objectives and 
content. Were the lab equipment and materials easy to set up and 
use by the students and teachers? The observer should note any 
safety problems that the equipir at and materials may cause. More- 
over, was the instruction time allotted for the laboratory activi- 
ties sufficient? Overall impressions of the lab activities would 
be welcome. 



viii. General 

Overall impressions of the classroom observation should be noted. 
What did the students like best about the unit? What did the stu- 
dents dislike most about the unit? Did the teacher or students 
appear to be interested? Were they bored? What parts of the unit 
did the students and teachers appear to find most interesting? 
Did the teacher or students become confused during the instruction 
period? If so, what caused the confusion? Any additional com- 
ments that the observer has about the classroom observation should 
be noted. 
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DESCRIPTION OF TOPICS AND SUBTOPICS 
(DOCUMENT/RECORD REVIEWS) 

I. Obiert j ves 

Included in the document/record review should be the reviewer's 
impressions of the un<t objective development process. 
Specifically, were the objectives developed by those knowledgeable 
about the subject matter needed by secondary vocational education 
students? Did the objective development process ensure that an 
appropriate coverage of unit objectives was included in the 
affective, psychomotor, and cognitive domains? Moreover, within 
each of these domains, were the objectives developed at an 
appropriate level of difficulty? Development procedures designed 
to clarify the meaning of each objective and the criteria used to 
determine what the student needs to know should be noted. 

II. Content 

The reviewer should examine content development procedures 
designed to ensure that unit content is presented in a logical 
order and at the correct level of difficulty and language level. 
What rationale did the developers follow to determine the adequacy 
of the amount of unit information? Reviewer impressions of each 
unit's clarity of directions, meaningf ulness of unit examples, and 
the number of examples should be noted. In sum, what procedures 
did the curriculum developer follow to ensure appropriate unit 
content? 

III. Learner Activities 

The reviewer should examine the learning activity development 
process, what steps were taken by the developer to ensure that 
learner activities are relevant to unit objectives and content? 
Documentation that provide evidence that the learner activities 
are appropriately sequenced, are of an appropriate number, and are 
an efficient use of time should be noted. Records that indicate 
problems encountered by learner activities should be noted as 
well. 

IV. Teacher 

Records pertaining to teacher background should be reviewed. Do 
the teacher*, have adequate knowledge to teach Principles of 
Technology? Moreover, what evidence exists that teachers make 
effective use r instructional techniques? 
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v. Posttest 



The reviewer should examine procedures used in developing posttest 
items. What steps did the developer take to ensure that posttest 
items covered important unit points P that items are clearly 
worded, and that items are fair? In sum, are the unit posttests 
valid representations of unit content? 



VI . Instructional Aids 

The reviewer should examine the instructional aid development 
process. Were steps taken by the developer to ensure that 
instructional aids are appropriace for unit objectives and 
content? Evidence that the instructional aids reinforce unit 
concepts and are interesting to the students and teachers should 
be noted. 



VII. Equipment and Materials 

The reviewer should ascertain whether the equipment and materials 
are appropriate for unit objectives and content. Specifically, 
records pertaining to difficulty of use, safety problems, 
availability, cost, construction complexity, and time required for 
effective instruction should be examined. 



VIII. General 

After reviewing documents and records pertaining to the Principles 
of Technology development process, the reviewer is encouraged to 
make general comments about the curriculum. For example, what did 
the reviewer like and dislike most about the curriculum? What 
parts of the units were boring or confusing? What is an 
appropriate amount of homework? The reviewer should feel free to 
make additional comments, specific or general, about the 
curriculum or the curriculum development process. 
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(SAMPLE) 
INTERVIEW NOTE SHEET 



SITE: 


INTERVIEWEE: 
NAME: 


DATE: 


TITLE: 


TIME: 


ORG • 


INTERVIEWER 


: STREET: 

CITY: 
*?IP : 

TELEPHONE : 
INTERVIEW NOTES 


CODE 


NARRATIVE 
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SITE: 

TEACHER: 
CLASS: _ 
OBSERVER: 
DATE: 



CLASSROOM OBSERVATIONS 



OBSERVATIONS (NARRATIVE) 



(This section should contain 
only the account of what is 
actually observed or said. No 
judgments or evaluations should 
be listed in this side of the 
observational record. ) 



Notes (Feelings or impres- 
sions about what was done 
or said) 



(This section should contain 
the observer's interpreta- 
tion of what was seen and/or 
heard. ) 



1 



SITE: 

TEACHER: 

CLASS: 

REVIEWER: 
DATE: 



Document/Record Title: 



CODE 


INFORMATION 
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The case study data must be stored in a format that makes it 
easily retrievable. The coding system should be one which (1) 
has the data easily available for analysis, (2) is the least time 
consuming, (3) is easy to implement, and (4) is cost effective. 

The coding system should be designed so that each piece of 
data will be classified. A scheme needs to be established for 
identifying the school, the evaluator collecting the information 
at the site, the type of person (teacher, etc.) being interviewed 
or observed, the kind of document/record reviewed, and the data 
collection procedure (interview, observation, etc.). Each 
interview, observation, or document/record reviewed should be 
assigned a sequential number. 

Each bit of information should be recorded on a separate 
index card or be retrievaole from a computer. These small pieces 
of data can then be shuffled around in order to look at the 
information from different perspectives and to detect emerging 
themes. 



Staff Training 

It is crucial that a training session be conducted for all 
staff to be involved in the case studies. The first step to 
designing staff training is to identify the needs of the staff. 
The following questions might be used to identify and prioritize 
staff training needs 



STAFF TRAINING NEEDS 

Rank Order Should staff training include: yes no 

b 1) Background information on the 

purpose of the case study? [ ] [ ] 

2) Clear definitions of terms and 

concepts to be used? [ ] [ ] 

3) Discussion of what to do before 

entering the site? [ ] [ ] 

4) Discussion of what to do when on 

site? [ ] [ ] 
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5) Procedures to be used for the case 

study? [ ] [ ] 

6) An explanation of the coding system? [ ] [ ] 

7) Discussion/review of interview, 
observation, and document/record 

review methods? [ ] [ ] 

8) Practice with interview, observation, 
and document/record review methods 
through structured activities (e.g., 

role playing) ? [ ] [ ] 

9) Practices with interview, observation, 
and document/ record reviews in the 

field? [ ] [ j 

10) Practice using the coding system? [ ] [ ] 

11) Opportunity to modify 

a) case study procedures? [ ] [ ] 

b) coding system? [ ] [ ] 

c) persons to be interviewed? [ ] [ ] 

12) [ ] [ ] 

13) [ ] [ ] 

14) [ ] [ ] 



A staff training session will probably cover a minimum of 
3 days. Following is a sample staff training agenda. 



SAMPLE STAFF TRAINING AGENDA 

DAY 1 

AM: Background and purpose of case studies 
Review of site selection process 
Review of schedrle and work assignments 
Discussion of logistics 
Discussion of topics and subtopics 

Discussion of coding system* including coding exercise 
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PM: Interviewing procedures and techniques 
Practice interviewing (role playing) 
Observation procedures and techniques 
Practice observing (role playing) 
Reviewing documents and records 
Exercise on document/record review 
Using extant data 



DAi 2 

AM: Practice interviewing, observing, and document/record 
reviews at sample site 

PM: Write up field notes and code informatio i 



DAY 3 

AM: Evaluate field activity 

Conduct data analysis to detect emerging trends, issues, 
problems 

PM: Review procedures ana rev "se, if needed 
Review of assignments 



Logistics of Fieldwork Operations and Pat;: Collection 

Following is a potpourri or do's and don'ts as one gets ready 
to enter the field. 

Scheduling , Interviews and observations should be scheduled 
well in advance of site visit. If this is accomplished by tele- 
phone there should be a follow-up confirming letter. Daily sched- 
ules should be developed, remembering that at least as much 
time is needed to write up notes and code information as is needed 
to collect it. 

Selecting interviewees, obs ervations, and document/records . 
Selection should be mad° by the evaluatoL3. Site personnel may 
make suggestions, but the final decisions should be made by the 
^valuators . 

Recording responses . Decisions should be made about whether 
written notes or tape recording is required. 

Participation . The primary concern iL to collect data. The 
amount of participation should be such that the evaluator fits 
comfortably into the setting without disturbing or interfering 
witn normal operations. 
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When in Rome * • « . Learn the language or jargon used at 

the site. This is essential if the ^valuator is to be able to 
pick up on insights and clues as to how people at the site view 
and define situations, events, and things. 

Interviewing suggestions. The following suggestions are made 
for conducting the interviews: 

o Use an open-ended intervi ewing techniq ue. 

To the extent possible, let the interviewee introduce what 

he/she considers relevant. The interviewer should set the 

context and then conduct the interview as a dialogue, not 

a question-by-question process of acquiring information 
f K om the interviewee. 

o Establish a context for i nterviewing , 

- Secure a private room for the interview. 

- Arsure confidentiality. 

- Try to avoid group interviews. 

- Be ready when the interviewee enters room. Have note 
pad, pens, recorder, handouts ready. 

- Set a tone that conveys the importance of the inter- 
view. 

- Keep the interview within prearranged time frame e 

o Use successful interviewi ng techniques - 

- Be a good listener 

- Pursue in detail: seek examples and illustrations. 

- Take notes rapidly. 

- Be attentive. 

- Seek clarification. 

- Keep questions brief. 

- Begin interview with simpler questions. 

- Do not breach confidentiality. 

o Potential Probes 

- Pause 

- Repetition of question or part of question 

- "What do you mean?" or "How do you mean?" 

- "Would you tell me more about your thinking on that?" 

- "What do you think?" or >? What do you expect?" 

- "Which would be closer to the way you feel?" or "Which 
would be closer?" 

- "Are there any other reasons why you feel that w?.y?" 
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o Neutral Prefaces 

- "Of course no one knows for sure ..." 

"Of course there are no right or wrong answers . . . " 
"Of course there are no righw or wrong answers; we're 
interested in people's opinions . . •" 

- "We're just interested in what you think ..." 

- "We all hope, but ..." 

- "Let me repeat the question ..." 
"Well, in general ..." 

- "Generally speaking ..." 

- "Overall ..." 

- "In the Principles of Technology project as a whole..." 

- "Yes, but ..." 

O Carefull y record rlaf* . 

- Take extensive notes and/or record interview. 
Expand and complete notes as soon as interview is 
completed. 

- Code notes. 

- Type and edit notes. 

Observations suggestions. The following suggestions are made 
conducting the classroom observations: 

o The observations should be of a nonparticipatory nature. 
The teacher and students should be aware that they are 
being observed, but the observer should not become a 
member of the group in the sense that he/she has a stake 
in the jroup outcomes. 

o Observation involves inspection and contemplation of what 
one sees pnd hears. This necessitates an understanding of 
the cont in which the Principles of Technology class 
operates and a recognition of the emotions and feelings 
shaping what one sees and hears. 

o Extensive notes should be taken during tht observation 
period. 

o Following the observations, the notes should be expanded 

and coded according to the topic and subtopic areas. 

o Be especially aware that the rights and privacj of obser- 
vees are not sacrificed. 
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PPCUment/rec.rd review sjigg fi&fc jojia . The followin 
tions are made for reviewing documents and records. 



g sugges- 



o Examples of records and documents that might be reviewed 
include audits, supervisory reports, classroom expenses, 
cests, test results, teacher and student evaluations of 
Principles of Technology units, letters, memoranda, 
newspaper articles, tapes of radio broadcasts about the 
class, editorials, teacher generated materials, and so 



o Documents and records irequentlv provide a valuable sen 
of the context in which a program or class operates. T 
should be reviewed prior to conducting observations and 
inte rviews. 

o "what is said", "How it is said", and "When it is said" 
are important considerations in all data collection, 
including document and record review. 

o The reviewer needs to be cognizant of data that are per- 
sonal, private, or classified. Caution must be exercis< 
so as not to create special ethical problems. 



Analyzing 

The evaluator in the field begins the data analysis process 
as soon as the first data are collected. The evaluator will be 




the evaluator reading through all of the data in order to become 
ramiiiar with the information obtained. This reading of all of 
the information collected is essential to confirm patterns and 
themes that emerged as the data were being collected and to iden- 
tify new patterns and themes. 
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Reporting the Finding s 



The individual (s) responsible for collecting the data at a 
given site should be responsible for writing the report for that 
site. The individual (s) who have been on-site will have a much 
better feel for the data and will be able to provide a better 
sense of reality. The report writer may find the following 
helpful : 



REPORTING THE FINDINGS 
Did I: Yes No 

1) Explain the background (e.g., purpose, 
limitations) . [ ] [ ] 

2) Describe the site completely. [ ] [ ] 

3) Use accurate, detailed, descriptions of 

actions and behaviors. [ ] [ ] 

4) Report word-for-word statements where 
appropriate. [ ] [ ] 

5) Report whether interaction with informants 
was effective and whether informants 

were reliable. [ ] [ ] 

6) Has data been used from all possible 
sources (e.g. , interviews, observations , 
record/document reviews, extant data, 

other)? [ ] [ ] 

7) Did evaluator's presence and/or actions 

create disruptive or unnatural conditions? [ ] [ ] 

8) Have evaluator's pre-formed opinions unduly 
influenced data interpretation? [ ] [ ] 

9) Do interpretations flow directly from 

the data? [ ] [ ] 



Enhancing case study rep ort utilization . Potential for 
utilization should be a concern throughout the design, implementa- 
tion, and reporting of the case study. Some suggestions for 
enhancing the probability of using case study findings are: 

o The people for whom the information is intended should 
be involved in the initial stages of the planning. It 
is essential that they be involved in clarifying what 
their information needs are. 

o Develop summary reports oriented to the needs of speci- 
fic audiences. 

o Conduct meetings with potential users so that evaluators 
and potential users can interact about various aspects 
of the report. 
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Unit # 

Principles of Technology Unit Evaluation 
Student Questionnaire 

Grade Sex Student ID # 

PURPOSE 

Your honest answers to the questions on this form will help improve this Principles of Technology unit for other 
students. Your responses to the following items are appreciated. 

DIRECTIONS 

Below ait a list of statements. Please read each statement and answer whether you strongly agree, agree, disagree, or strongly 
disagree by checking the appropriate box. Place a check mark [V] in only one box for each statement 



Strongly Strongly 
Agree Agree Disagree Disagree 



UNIT OBJECTIVES 
For this unit: 



1. the objectives helped mt understand what I was 

supposed to learn 


[] 


[] 


[] 


[] 


2. I'm glad the objectives are printed in the written 


[] 


[] 


[] 


[] 


3. I used the objectives to guide me through the 

material 


[] 


[] 


[] 


I] 


General comments on Unit Objectives: 










UNIT CONTENT 










Hie content of this unit: 












[] 


[] 


[] 


[) 


5. gave examples helpful in understanding 

concepts 


[] 


[] 


[] 


[] 


6. was too difficult for me to understand 


[] 


[] 


[] 


[] 


7. will probably be useful in future jobs 


(] 


[] 


[] 


[] 



General comments on Unit Content: 
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Strongly Strongly 
Agree Agree Disagree Disagree 



UNIT VIDEO 

The video programs for this unit: 

8. helped me to better understand the text 

material U [1 N U 

9. were interesting [ ] [1 N U 

10. used easy to understand graphics [ ] [ ] [ 1 □ 

11. helped me to achieve the unit objectives [] [] [] □ 

General Comments on Unit Video: 



UNIT TEXT 

The unit text materials: 

12. helped me to achieve the unit objectives [] [] U (1 

13. will be a useful reference after taking the 

course N U [1 M 

14. used language difficult for me to understand. [] [] U U 

1 5. had enough examples to help me understand the 

important concepts [ ] [1 [ 1 [1 

16. helped me to understand the unit concepts [] [] M N 

General comments on Unit Text: 



LABORATORY ACTIVITIES 

The unit lab activities: 

17. helped me achieve the unit objectives [ ] [ ] [ ] N 

18. were difficult [] [1 N M 

19. time periods were not long enough to complete 

the work [] N M N 
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Strongly Strongly 
Agree Agree Disagree Disagree 





11 


[] 


[] 


(1 


General comments on Laboratory Activities: 










UNIT DIFFICULTY 










ror mis unit. 










21. some of the things we were expected to learn 


[1 


[] 


[] 


[] 




r i 


r i 


U 


r i 
I J 


General comments on Unit Difficulty: 










STUDENT SATISFACTION 










For this unit 










23. I usually had a sense of satisfaction after leaving 

class each day 


[] 


[] 


[] 


u 


24. I did not like coming to class 


[] 


[] 


[] 


[] 


25. I would recommend it to my friends 


[] 


[] 


[] 


[] 



General comments on Student Satisfaction: 
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Unit # 



Teacher # 



PURPOSE 



Principles of Technology Unit Evaluation 
Teacher Questionnaire 



The primary purpose of this questionnaire is to provide teacher information for facilitating improvement of the Principles 
of Technology curriculum. Specifically, this form is designed to nssesi the appropriateness of Principles of 
Technology unit objectives and the appropriateness of instructional activities, resources, curriculum content, and 
instructional planning needed to accomplish these objectives. 

DIRECTIONS 

Below are a list of statements. Please read each statement and indicate whether you strongly agree, agree, disagree, or strongly 
disagree by checking the appropriate box. Place a check mark [V] in only one box for each statement 



Strongly Strongly 
Agree Agree Disagree Disagree 



UNIT OBJECTIVES 
The unit objectives: 



1. assist me in assessing student progress 


[] 


[] 


[] 


[] 


2. don't reflect what the unit is designed to teach 


[] 


[] 


[] 


[] 


3. help me determine what to teach 


[J 


[] 


[] 


[] 


4. are presented in logical order 


[) 


[] 


[] 


[] 



General comments on Objectives: 



UNIT INSTRUCTIONAL ACTIVITIES 

The unit instructional activities: 

5. are related to unit objectives 

6. are presented in the correct sequence , 

7. require a reading level that is too difficult for most 
students 

General comments on Instructional Activities: 



t) 
[) 

[] 



[] 
[] 



[] 
[] 

[] 



[3 

[] 

(1 
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19. Based on your experiences, do you think the 6-day plan, per sub-unit, of 50- minute class sessions 
is realistic for this unit? 



If no, please explain: 

21. What, if anything, caused you the most problems in teaching this unit? 

22. What did you like the most in teaching this unit? 
General comments on Perceptions of Instructional Planning: 

23. Did you teach this unit on consecutive days for 26 days? 

NYes []No 

If no, what pattern did you use (for example, 3 days a week)? 

24. How much time per class session did you devote to this unit? 

[ ] 50 minutes or less [] 61-90 minutes 

[ ] 51-60 minutes [ ] 91+ minutes 

25. Did you combine any of the unit s classes into one session (for example, teach classes CI and C2 

in one session)? 

[ ] Yes [ ] No 

If yes, which classes did you combine? 



20. 



[ ] Yes, definitely ( ] No, probably not 

[ ] Yes, probably [ j No, definitely not 

Overall, did you feel comfortable teaching the materials in this unit? 



[ ] Yes, very comfortable 
[ j Yes, sort of comfortable 



[ ] No, not very comfortable 
[] No, not at all comfortable 
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Stronglj Strongly 
Agree Agree Disagree Disagree 



UNIT CONTENT 
The unit content: 

8. was too detailed tl 

9. was presented at the proper level of difficulty [ ] 

10. contained examples that were helpful to students in 

their understanding of unit concepts [ 1 

11. contained enough examples U 

12. provided enough summaries of important points [ ] 

13. provided information that will be useful for students 

in their future employment [ ] 

General comments on Unit Content: 
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PERCEPTIONS OF INSTRUCTIONAL PLANNING 

14. On average, how many hours did you spend preparing to teach each lesson in this unit? 

15. Approximately how much time, on average, did you expect students to spend on homework each day 

for this unit? 

[ ] None 1 1 About one hour 

[ ] About 15 minutes [ 1 Two hours or more 

[ ] About half an hour 

16. What percentage of students typically completed your homework assignments for this unit? 

[ ] Fewer than 25% [ ] 50-74% 

[ ] 26-49% 1 1 75 * ™ d rto^ 

17. What percentage of students typically completed your homework assignments for other courses that 

you teach? 

[] Fewer than 25% [] 50-74% 

[] 26-49% [] 75% and above 

18. Do you feel the Teacher's Guide material provided you with enough information to help you teach 

the unit? 

[ ] Definitely [ ] Probably [ ] Probably not [ ] Definitely not 
If not, what should be added to the guide to make it more useful? 
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26. Were there any special circumstances which, in your opinion, may have influenced the pre-or posttest 
scores? (e.g. lack of time; faulty lab equipment; school holidays; fire drill in the middle of the exam) 



General Comments: 
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Unit # 

Teacher ID # 

Principles of Technology 
Unit Daily Log 

PURPOSE 

The purpose of the Unit Daily Log is to provide a regular source of teacher information for improving the Principles of 
Technology curriculum. 

DIRECTIONS 

The left column on the following charts lists each activity for this particul m unit Since there are no materials specifically 
for the sub-unit review classes, these classes have not been listed in the chart 

For columns 1-5 on the attached charts: 

1. Circle T (yes) or "N" (no): Were the readings, labs, or videos appropriate (e.g., grade level, sufficient quantity 

of material) for your students? If not, specify modifications in column S. 

2. Circle T (yes) or "N" (no): Were you able to cover the content in the 50-minute time period? 

3. Briefly describe any errors or inaccuracies in the material. 

4. Briefly describe any problems you had in managing the material. Among others, this may include problems in 

coordinating lab rotations, lab set-up, and maintaining student interest 

5. Briefly list suggestions for modifying the material Describe "teaching tips" for future teachers. 

We recommend that you take a few minutes each day to complete the charts. If you need more space for comments, use the 
back of the charts and/or attach additional pages. 



9 

ERLC 



128 



133 



[1] 

Appropriate 
u.;it for students? 


12] 

Completed in 
50-60 minutes? 


[3] 

Did you discover any errors or 
inaccuracies? If so, please 
specify in Che appropriate 
spaces below. 


[4] 

Did you have any management 
problems? If so, please specify 
in the appropriate spaces below. 


[5] 

Do you have any suggested mod- 
ifications or general comments? 
If so, please specify in the appropriate 
spaces below. 


OVERVIEW 

Video Y N 


X 








CO Y N 


Y N 








WAVE CHARAC- 
TERISTICS 

Video Y N 


^ > 








CI Y N 


Y N 








C2 Y N 


Y N 








Math Lab Y N 


Y N 








Lab 9-1 Y N 


Y N 








Lab 9-2 Y N 


Y N 
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[1] 

Appropriate 
Unit for students? 


[2] 

Completed in 
5040 minutes? 


[3] 

Did you discover any errors or 
inaccuracies? If so, please 
specify in the appropriate 
spaces below. 

. 


[4] 

Did you have any management 
problems? If so, please specify 
in the appropriate spaces below. 


[51 

Do you have any suggested mod- 
ifications or general comments? 
If so, please specify in the appropriate 
spaces below. 


WAVE APPLI- 
CATIONS 

Video Y N 


A 









CI Y N 


Y N 








C2 Y N 


Y N 








MathLa Y N 


Y N 








Lab 9-3 Y N 


Y N 








Lab9-4 Y N 


Y N 








Summary 

Video Y N 
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SUGGESTED AUDIT INTERVIEW QUESTIONS 
Key: 

SQ= Student Questionnaire 

TQ= Teacher Questionnaire 

UDL= Unit Daily Log 

1= Interviews 

CO= Classroom Observations 

DR= Document Reviews 



S= Student Interview 
T= Teacher Interview 
SA= School Administrator 
Interview 

Agency for Instructional 
Technology Representative 
view 



U= Utility 
F= Feasibility 
P= Propriety 
A= Accuracy 
DI= Developer Implementation 
Example: U.2= Clarity of Information 
F.l= Practical Procedures 
P. 3= Balanced Reporting 



UTILITY (U) 

1. Information Scoue and Selection (T. AIT) 



A. To what extent did the SQ, TQ, UDL, I, CO, DR address the 
information needs of the curriculum developers? 

B. Was the information collected by the SQ, TQ, UDL, I, CO, 
DR sufficient to support a judgment of merit or worth? If 
so, to what extent? If not, what were the shortcomings? 

C. To what extent was the information collected by the SQ, 
TQ, UDL, I CO, DR useful? To what extent was the infor- 
mation collected redundant or meaningless? 

D. To what extent were the formative evaluation questions 
answered by the information collected from the six data 
collection activities? 



2. Clarity of Informa tion (ATT) 

A. To what extent was the information collected by SQ, TQ, 
UDL, I, CO, DR understandable to the curriculum develop- 
ers? 



B. To what extent did the information collected by SQ, TQ, 

UDL, I, CO, DR respond clearly and without generalities to 
the formative evaluation objectives? 
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C. To what extent did the information collected by SQ, TQ, 
UDL, I, CO, DR provide a firm foundation for formative 
evaluation conclusions and recommendations? 

D. To what extent was the information collected by the six 
data collection activities characterized by conciseness 
and logical development? 

E. In sum, how well did the information collected by the six 
data collection activities communicate to the formative 
evaluation audiences? 

3, Timeliness (AIT) 

A. To what extent was the information collected by SQ, TQ, 
UDL, I, CO, DR, received by the curriculum developers on 
scheduled timeline dates? 

B. How many curriculum revisions were delayed as a result of 
late formative evaluation information collected by SQ, TQ, 
UDL, I, CO, DR? What were the causes of these delays? 

C. In sum, to what extent were the curriculum developer's 
most critical information needs met on time? 



FEASIBILITY (F) 

1. Practical Procedures (T, A, AIT) 

A, To what extent were the SQ, TQ, UDL, I, CO, DR sugges- 
tions made by National Center staff for strengthening 
the formative evaluation realistic given time and 
financial constraints inherent in the formative 
evaluation project? 

B. Were the SQ, TQ, UDL, I, CO, Dr evaluation procedures a 
practical approach to the formative evaluation of the 
Principles of Technology curriculum? To what extent were 
school and classroom disruptions minimized? 

2. Political Viability f ATT) 

A. To what extent were the evaluation procedures politically 
viable given the various interest groups and stakeholders 
of the Principles of Technology curriculum project? 

B. Did any political conflicts erupt over the evaluation 
effort? If so, how were these conflicts handled? What 
impact did these conflicts have on achieving the purpose 
of the evaluation? 
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Cost Effectiveness (ATT) 



A. To what extent did the evaluation procedure information of 
sufficient value to justify the expense of tne evaluation? 

B. Were there alternative evaluation approaches that would 
produce more useful information at the same or less cost? 
If so, please describe. 

C. Was the second year evaluation effort begun without a 
commitment of sufficient resourcas to ensure completion? 
If so, what data collection activities had to be 
compromised? 

D. For the information needed/ to what extent was the 
evaluation conducted as economically as possible? 



PROPRIETY (P) 
Rights of Human Subjects (S. T. A, AIT) 

A. To what extent were the rights and welfare of the 
participants who participated in the evaluation respected 
and protected? 

B. To what extent were legal, ethical, and common sense 
considerations followed? 

Conflict s of Interests fT. A, AIT) 

A. To what extent were conflicts of interest in the 

evaluation effort avoided? If conflicts of interest 
arose, how were they dealt with so that the evaluation 
process and results were not compromised? 

Balanced Reportin g (ATT) 

A. To what extent was the evaluation complete and fair in its 
presentation of strengths and weaknesses? Were both 
negative and positive aspects of the curriculum evaluation 
reported? Give examples. 

B. To what extent were evaluation findings that might prove 
embarrassing to partisan interest groups omitted from 
formative evaluation findings? In other words, what 
strengths and weaknesses were manipulated to please 
partisan interest groups? 
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ACCURACY (AC) 
Clearly Identified object f AIT ) 

A. To what extent was the object of the formative evaluation 
clearly identified? 

B. Given project constraints, to what extent was the object 
of the evaluation described realistically? 

Purposes and Pro cedures Explicated (S, T, A. AIT) 

A. To what extent were the purposes and procedures of the 
formative evaluation understood by you? 

B. Were you clear on the objectives of the evaluation? 

C. To what extent were procedures on how information was 
collected, organized, analyzed, and reported clearly 
delineated? 

D. If the evaluation were to be replicated, were purposes and 
procedures described in sufficient detail so that other 
evaluators could conduct a similar evaluation in other 
settings. If not, what do you suggest for improving the 
purposes and procedures? 



DEVELOPER IMPLEMENTATION (DI) 
Preparation (S. T. A, att) 

A. To what extent were you (S, T, A, AIT) prepared for the 
formative evaluation? 

B. To what extent were you (teachers) prepared to implement 
the evaluation instruments? 
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